Point-to-multipoint high definition multimedia transmitter and receiver

ABSTRACT

A high definition video transmitter and receiver are disclosed. The transmitter provides high definition video to a one-point receiver or to multipoint receivers. The transmission network is asynchronous and the receiver re-synchronizes the video. The transmission can be wired or wireless.

This application claims priority to U.S. Patent Application No.60/842,706, filed Sep. 7, 2006, the entire contents of which isincorporated herein by reference.

FIELD OF THE INVENTION

This relates to point to multipoint transmission and reception of highdefinition video communications.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is better understood by reading the following detaileddescription with reference to the accompanying drawings in which:

FIG. 1 is a schematic representation of a system according to an exampleembodiment of the present invention;

FIGS. 2-3 are perspective views of an example transmitter;

FIG. 4 is a perspective view of an example receiver;

FIGS. 5-6 are schematics of an example software architecture;

FIGS. 7 and 8 are schematics of example system architectures for exampletransmitter and receiver, respectively;

FIGS. 9 and 10 are charts showing data flow for the transmitter andreceiver, respectively.

FIG. 11 is a schematic representation of the transmit frame buffers;

FIG. 12 is a schematic format for the transmitter video frame buffers;

FIG. 13 is a schematic format for the receiver video frame buffers;

FIG. 14 is a timing diagram of the transmitter and receiver clocks;

FIG. 15 is a timing diagram of the normalized receiver clock and offset;

FIG. 16 is a timing diagram of the measured offset between transmitterand receiver clocks;

FIG. 17 is a timing diagram of the estimated transmitter clock at thereceiver;

FIG. 18 is a timing diagram of a comparison between estimated and actualtransmitter clocks;

FIG. 19 is a schematic representation of the error caused by phasedifferences among hardware clocks;

FIG. 20 is a schematic of the encode video path;

FIG. 21 is a schematic of the decode video path;

FIG. 22 is a timing diagram of the interrupt timing;

FIG. 23 is a schematic representation of an example main processorcommunications sequence;

FIGS. 24-25 are diagrams of example display update sequences;

FIG. 26 is a block diagram of example web user interface system;

FIG. 27 is a flash memory map;

FIGS. 28-31 are video packet formats;

FIGS. 32-33 are audio packet formats;

FIG. 34 is an adaptive synchronization algorithm;

FIG. 35 is a block diagram of the group play back clock;

FIG. 36 is a block diagram of an example infrared remote extender;

FIG. 37 is a electrical schematic of the main processor board circuit;

FIGS. 38-41 are example main processor memory maps;

FIG. 42 is a set up for the expansion bus from the main processor to theexpansion connector;

FIG. 43 is a set up for the expansion bus from the main processor to theFPGA programming;

FIG. 44 is a schematic representation of three PCI slot specifications;

FIG. 45 is a schematic representation of four serial portspecifications;

FIG. 46 is an example main processor board clock tree;

FIG. 47 is an example main processor board clock register set up;

FIG. 48 is a table of GPIO pin set ups for the main processor;

FIG. 49 is a table of boot options as defined by the main processorboard;

FIG. 50 is a table of user defined values for boot options;

FIGS. 51A-B together are a table of expansion board connections;

FIG. 52 is a model of an example audio visual communication protocol;

FIG. 53 is an illustration of example A/V planes and layers of theexample system;

FIG. 54 is a schematic of example ALIP links;

FIGS. 55-61 are example message formats for ALIP messages;

FIGS. 62-66 are example scenarios of ALIP message usages;

FIG. 67 is a schematic of example ASIP sessions;

FIGS. 68-86 are example message formats for ASIP sessions;

FIGS. 87-91 are example scenarios of ASIP sessions;

FIG. 92 is a schematic representation of example RTP audio and videosessions; and

FIGS. 93-99 are example RTP message formats.

DETAILED DESCRIPTION OF THE INVENTION & BACKGROUND AND SUMMARY

Certain commercial specifications and protocols are relevant to thepresent description, each of which is incorporated herein by reference.They include:

(1) “High-Definition Multimedia Interface Specification, Version 1.2”;

(2) “High-bandwidth Digital Content Protection System”;

(3) “RTP: A Transport Protocol for Real-Time Applications”;

(4) “RTP Payload Format for JPEG 2000 Video Streams”;

(5) “The Secure Real-time Transport Protocol”; and

(6) “Avocent Audio Visual Protocol Specification,” a copy of which isincluded in U.S. Patent Application No. 60/842,706, filed Sep. 7, 2006.Certain papers and books provide background material for the presentdescription, the teachings of which will be assumed to be known to thereader. They include:

(1) O'Reilly, “802.11 Wireless Networks The Definitive Guide,”Sebastopol, Calif., 2005;

(2) Castro, Elizabeth, “HTML for the World Wide Web, Fifth Edition, withXHTML and CSS: Visual QuickStart Guide,” Berkeley: Peachpit Press, 2003;

(3) Castro, Elizabeth, “Perl and CGI for the World Wide Web, SecondEdition: Visual QuickStart Guide,” Berkley: Peachpit Press, 2001;

(4) “Intrastream Synchronization for Continuous Media Streams: A Surveyof Playout Schedulers”;

(5) “Multipoint Multimedia Teleconference System with AdaptiveSynchronization”;

(6) “The Design of the OpenBSD Cryptographic Framework”; and

(7) “Survey of Error Recovery Techniques for IP-Based Audio-VisualMulticast Applications.”

Some manuals and documents are relevant to the operation of the mainprocessor board. Each of them are incorporated herein by reference, andinclude:

Certain manuals and datasheets describe in more detail various aspectsof the present description, each of which are incorporated wherein byreference. They include:

For the processor:

(1) Intel IXP45X and Intel IXP46X Product Line of Network ProcessorsDeveloper's Manual (located athttp://www.intel.com/design/network/products/npfamily/docs/ixp4xx.htm

(2) For the processor, Intel IXP45X and Intel IXP46X Product Line ofNetwork Processors Data Sheet (located at same link immediately above)

For the audio visual subsystem:

(3) HDMI Display Interface—Preliminary Technical Data—AD9398

(4) 800 Mbit High Performance HDMI/DVI Transmitter—Preliminary TechnicalData—AD9889

(5) ADV7401 Integrated Multi-Format SDTV/HDTV Video Decoder and RGBGraphics Digitizer

(6) Multiformat 216 MHz Video Encoder with Six NSV 12-BitDACs—ADV7320/ADV7321

(7) JPEG2000 Video Codec ADV202

(8) ADV202 JPEG2000 Video Processor User's Guide

(9) Getting Started with the ADV202 Programming Guide

(10) ADV202 Output Formats and Attribute Data Format

(11) Using the ADV202 in a Multi-chip Application

(12) Front Panel Processor (LPC2103) For the Radio Chips:

(13) AR5002 STA Programmer's Guide

(14) Boa Documentation;

For the DDR:

(15) Micron Double Data Rate (DDR) SDRAM Data Sheet

The documents described above are incorporated by reference.

The present description relates to a multipoint extender (wired orwireless) for transmitting high-definition content from one source toone or more destinations incorporating High-Definition MultimediaInterface (HDMI) technology. In its preferred embodiment, the system canwirelessly transmit high resolution computer graphics, high-definitionvideo, stereo audio, and control data to eight wireless or LAN connectedreceivers. The multipoint system includes a transmitter and receiversthat provide high-definition media support suitable for suchapplications as professional audio-visual applications. Audio-videosynchronization is provided at each display. All wirelessly connectedreceivers remain synchronized with each other. Interchangeable modulescan be added for analog signals including composite video, componentvideo, computer graphics, or digital HDMI/DVI signals.

In FIG. 1, the system 10 provides a wire replacement for a HighDefinition Multimedia Interface (in the art sometimes abbreviated HDMI,or simply HD) source connecting to an HDMI display. The system iscomposed of a transmitter 11 and one or more receivers 12. Thetransmitter will support communication to up to 8 receiver units. Eachreceiver will utilize a virtual clock that is synchronized by thetransmitter therefore allowing each receiver to be in sync with eachother receiver.

The audio and video data is encoded by the transmitter 11 that isconnected to the HDMI source 13 and transmits the data to a receiver 12.The receiver units 12 can be connected to respective HDMI displaydevices 14. The transmitter 11 and receiver 12 are based around JPEG2000codecs that will support the compression of High Definition Video alongwith standard definition content. JPEG2000 codecs are not new and arewell-known video transmission protocols to the artisan. It is notnecessary to repeat their details here.

Video, audio and control data (such as RS-232, infra-red, HDMI, orConsumer Electronics Control codes), which are supported at the hardwarelevel but could be supported either by software or firmware level, areinput into the transmitter 11. In the transmitter, the video & audiodata is converted to the digital domain, time stamped, compressed,encrypted, packetized, and transmitted over a 10/100 bit Ethernet linkor over a wireless link to the receiver. The order in which encryptionoccurs is not a requirement.

In an example embodiment, the above-described control data is notinterpreted at the transmitter, but is forwarded as-is.

At the other end, the receiver 12 decrypts, decompresses,converts/manipulates the data back into the desired format, and outputsto the connected HDMI-supported audio/video equipment (such as videodisplay 14). The control data that is received by the receiver 12 allowsfor several methods to control the external A/V equipment. In someexamples, this is done through RS232 serial, Infra-Red (IR)/IR-Blasterand CEC commands.

The present set of transmitter 11 and receiver 12 systems will allowdifferent A/V modules to be installed, such as the HDMI/Euro-Blockmodule.

The block diagrams and detailed design information for the HW appliancescan be found below.

The transmitter is designed to accept an input source 13, such as eithera HDMI source (Audio & Video) or one DVI source, either of which willaccept PC graphics or video. The rear of the Transmitter is shown inFIG. 3 and includes an 10/100 Ethernet port 17, RS-232 ports 24, and USBClient port 19.

The front of the transmitter is shown in FIG. 2 and includes a VacuumFluorescent Display (VFD) 15 or similar display to communicate to theuser the state of the device along with pertinent information, InfraRadreceiver and blaster pair 16, five control buttons (UP, DOWN, LEFT,RIGHT, and SELECT) 23, and a USB host port 18 to supply power for anoptional light.) The transmitter 11 also includes a single lane PCIexpress port 20, and ports for HDMI or DVI source video 21. Ports 22receive balanced, stereo audio input.

The receiver 12 is shown in more detail in FIGS. 4 (front) and 5 (back).It includes an HDMI video connector 25, Ethernet port 26, USB host port27, USB client port 28, RS232 port 29, and InfraRed transmitter andreceiver pair 30. The I/R pair 30 (as well as the I/R pair 16 on thetransmitter) support signals at 38 KHz signal+/−2 KHz window.

In general, the system incorporates a High Definition source or standarddefinition source connected to the transmitter 11 (FIG. 1) via an HDMIconnection to the HDMI port 21. A distance from the transmitter (whichcould be in excess of 100 feet) are N number (where N<=8 in thepreferred embodiment but more or less as appropriate) of receivers 12each connected via HDMI to a respective display device 14. Wirelesstransmission from the transmitter to the receivers is per 802.1certified radios.

HD-JPEG2000 is employed to compress the video and can run at,optionally, 20 MBps to meet commercial quality requirements.

The processor subsystem of the receiver 12 is based on a PCI Busarchitecture, using, for example, an Intel IXP455 processor with DDRmemory. The JPEG2000 codecs and WIFI radio communicate via the PCI bus.For that reason, the transmitter includes an 802.11a WIFI mini-PCI radiomounted on a PCI bus. The processor subsystem design allows use of theLinux OS (Kernel Version 2.6.x) which takes advantage of an OS with PCIsupport, network stacks, USB stacks, Radio drivers and an off the shelfWeb server.

The video codecs interface to the PCI bus. A PCI card in the receiverand transmitter supports the video processing circuits, the codec's andthe digital front ends. This will allow the front ends, video and audiocodec's to be tested prior to the being fully implemented in HW.

An FPGA optimizes the interface between the digital interface and themicroprocessor (uP). The FPGA handles (1) Configuration of the digitalfront ends, (2) audio time stamping, (3) video time stamping, and (4)configuration and read/write of compressed video data to and from thePCI bus.

Firmware on the transmitter 11 provides a number of functions. Thefirmware assigns each unit an electronic product identification numberthat is programmed at production time and cannot be modified by theuser. It assigns each unit with an electronic OEM identification numberthat is programmed at production time and cannot be modified by theuser. Each unit also has a country code stored in nonvolatile memory.

Each unit has a unique MAC address for each Wireless port (radio) in thedevice. The MAC address is programmed by the radio manufacturer andcannot be changed at production time or by the user. Each unit also hasa unique MAC address for the Ethernet port in the device that isprogrammed at production time and cannot be modified by the user. Forthe purpose of management, components can be identified by theirEthernet MAC address.

New operational firmware can be received via the RJ45 connection port.In the event of an interrupted or corrupted transfer, the unit remainssufficiently operational to communicate and receive a valid firmwareapplication. Alternatively, upgrades to firmware can be made via thewireless link. A firmware upgrade operation for the transmitter 11ideally contains the upgrade code for all processors and sub-processorscontained within the transmitter 11, rather than upgrading the parts ofthe transmitter individually.

The firmware also loads a test image for use in testing videotransmission to a receiver 12 and a test tone for use in testing audiotransmission to the receiver 12.

The firmware also allows the transmitter 11 to respond to ICMP pings.

The system is configured by the use of web pages. The user can configurethe system by the use of an intuitive menu system on multiple web pages.

The factory default configuration will set the IP and subnet and a givenpassword. The user will need to connect the PCs Ethernet port to thedevice. The user will be required to setup their PC to a valid IPaddress (other than the factory default IP address) per the subnet mask.The user will then enter the IP address of the unit into their webbrowser the follow the instructions for configuring the device(s).Examples of the type of information that will be provided andconfigurable are described below:

A. Transmitter Management Software—Web Pages

The Transmitter unit will allow configuration of the system by the userof a series of web pages. Information regarding specific modes orconfiguration will be consolidated onto specific web pages however theflow of the pages will be architected to facilitate easy configurationof the system. The information below is an example of the type ofinformation that can be displayed or be allowed to be input by the user.

-   -   Status Page        -   Display the Transmitter's name        -   Display the Transmitter's unique ID        -   Display the Receivers in range, signal strength and their            status        -   Display who is the master Receiver        -   Display the 802.11a Wireless channel being utilized.        -   Display country code    -   Version Page        -   Display AV subsystem firmware version        -   Display Front Panel firmware version        -   Display system hardware version        -   Display front panel hardware version        -   Display Transmitter ID        -   Display Transmitters name        -   Display the receivers joined the system and their:            -   IDs            -   Names            -   FW versions    -   Setup Page        -   Set the Transmitter's name        -   Set A/V connection type (Wired, Wireless)        -   Set IP Address of the Ethernet connection        -   Set Subnet mask of the Ethernet connection        -   Set the video quality        -   Set which Receiver will be part of the A/V system by            entering their unique ID        -   Set which Receiver will be the Master Receiver.        -   Set wireless Channel to utilize        -   Set transmit power        -   Enable/Disable            -   IR Blaster            -   RS-232            -   Wireless Radio

B. Receiver Management Software—Webpages

The Receiver unit will allow configuration of the system by the use of aseries of web pages. The information below is an example of the type ofinformation that can be displayed or be allowed to be input by the user.

-   -   Status Page        -   Display the Receiver's name        -   Display the Receiver's unique ID        -   Display the system's Transmitter's Name, ID, signal strength            and Status (Connected or not connected)        -   Display the 802.11a Wireless channel being utilized.        -   Display country code    -   Version Page        -   Display AV subsystem firmware version        -   Display system hardware version        -   Display Receivers ID        -   Display Receivers name    -   Setup Page        -   Set the Receivers name        -   Set A/V connection type (Wired, Wireless)        -   Set IP Address of the Ethernet connection        -   Set Subnet mask of the Ethernet connection        -   Set the ID of the Transmitter that the Receiver will            communicate with.        -   Set transmit power        -   Enable/Disable            -   IR Blaster            -   RS-232            -   Wireless Radio

Within a system 10, a single receiver 12 of the set of receivers (suchas are shown in FIG. 1) is designed as the master receiver. Datareceived by the serial port of the transmitter 11 is passed unmodifiedto the master receiver 12 which sends it out its RS232 port at the samedata rate as received by the transmitter 11. Data received by the serialport of the Master Receiver will be passed unmodified to the Transmitterwhich will send it out its RS232 port at the same data rate as receivedby the Master Receiver.

IR data received by the transmitter 11 is sent unmodified out thetransmitter IR blaster port 16 (unless disabled) as well as the masterreceiver IR blaster port 30. IR data received by the Master Receiverwill be sent unmodified out the Master Receiver IR (unless disabled)blaster port as well as the Transmitter IR blaster port.

A number of transmitters should be able to operate within a set radius(such as 200 feet) from each other without interfering with each other,although directional antennas may be required depending on the topologyspecified. Higher density operations may require a wired connectionbetween transmitter and receiver or the use of directional antennas andattenuation material.

Video information is received at the transmitter 11, encrypted, and sentby wireless, wireline or webpage to the appropriate receiver 12. ClosedCaptioning information located in lines 5-21 of active video inserted atan analog input of the transmitter are recoverable at the output of thereceiver.

For audio information, if a load is present on the Euro-block connectorof the transmitter 11, then audio is sampled from that input rather thanthe HDMI stream. The receiver 12 then outputs video on the HDMIconnector and Audio on Euro-blocks and/or RCA connectors. Audio receivedby the transmitter 11 on the HDMI stream is not transcoded by thetransmitter 11. Rather, valid HDMI audio is received by the transmitter11 and simply output to the receiver 12.

Receivers 12 in the system 10 are synchronized so (1) a viewer cannotperceive audio echoes from the display devices, and (2) a viewer cannotperceive that the video frames are not synchronized with the audio.

A protocol that typically causes an appliance to act as the ConsumerElectronics Control (CEC) root device will not in this system connectthe CEC line to any HDMI output. The transmitter 11 is the CEC root. Thetransmitter 11 generates the CEC Physical address of all source devicesconnected to it as per the HDMI specification, by appending a portnumber onto its own physical address and placing that value in theExtended Display Identification Data (EDID) for that port.

The system 10 acts as a HDMI Repeater/Distributor as defined in the HDMIspecification.

The user interface for configuring the transmitter 11 is via a web-based(http) browser. The user interface permits the user to:

-   -   Set the transmitter's name;    -   Set the Wireless Channel used for A/V stream;    -   Set the A/V connection Type (Wired, Wireless);    -   Set the IP Address of the Ethernet connection;    -   Set the Subnet mask of the Ethernet connection;    -   Set the Gateway for the Ethernet connection;    -   Display the version information (such as front panel firmware,        FGPA Version, hardware version, etc.);    -   Transmit an internal Test Signal;    -   Set the video quality level;    -   Set which Receiver unit(s) will be part of the A/V system by        entering their Unique ID (I.e. the MAC address);    -   Set which Receiver will be the Master Receiver;    -   Set transmit power below the regions max;    -   Allow enabling/disabling the IR Blaster (to prevent local        loopback);    -   Allow enabling/disabling the RS-232 pass through;    -   Set the RS-232 parameters;    -   Enter a password, before changing settings;    -   Set a password; and    -   Restore factory defaults.

The front panel operation is also governed by firmware that provides forseveral modes of displaying data. The signal strength mode displays thesignal strength of a single Receiver unit's transmission at a time. Whenin signal strength mode, the display periodically changes which Receiverunit's signal strength is displayed.

The error rate mode of display will display an indication of the errorrate of the communication channel for a single Receiver unit at a time.When in error rate mode, the display periodically changes which Receiverunit's error rate is displayed.

The channel mode of display will display the wireless channel beingutilized if the AV data stream is using wireless mode.

Finally, in the address mode of display, the IP address of thetransmitter 11 will be displayed on the VFD display.

On the receiver side, each receiver 12 supports DDC per the HDMIspecification. Like the transmitter 11, the receiver 12 has an 802.11aWIFI mini-PCI radio mounted on a mini-PCI bus. Like the transmitter,802.11a diversity antennae support wireless communications.

If a load is present on the Euro-block and/or RCA connector, then audiowill be routed to this output and not the HDMI stream.

Each receiver 12 can be connected or disconnected without causing damageto any other device in the system 10. This goal (hot-plugability) issatisfied in hardware design. When hot-plugging, all receivers 12,transmitter 11, and servers, mice, keyboards, monitors, and othernetwork devices remain uncompromised.

The receiver is controlled by an FPGA running firmware. The firmwareassigns each receiver with an electronic product identification numberthat is programmed at production time and cannot be modified by theuser. Each receiver also has an electronic OEM identification numberthat is programmed at production time and cannot be modified by theuser. Each receiver has a country code stored in nonvolatile memory.Each receiver has a unique MAC address for each Wireless port (radio) inthe device. Each unit has a unique MAC address for each Ethernet port inthe device that is programmed at production time and cannot be modifiedby the user. For the purpose of management, components are identified bytheir Ethernet MAC address. The firmware allows the receiver to respondto ICMP pings.

New operational firmware is received by the receiver via an EthernetRJ45 connection port. In the event of an interrupted or corruptedtransfer, the product shall remain sufficiently operational tocommunicate and receive a valid firmware application. A firmware upgradeoperation of the receiver contains the upgrade code for all processorsand sub-processors contained within the receiver, so there is no need toupgrade the parts of the receiver individually. Firmware upgrades canalso be done by the wireless link.

The user interface for configuring the receiver 12 is via a web-based(http) browser. The user interface permits the user to:

-   -   Set the receiver's name;    -   Set the IP Address of the Ethernet connection;    -   Set the Subnet mask of the Ethernet connection;    -   Set the Gateway for the Ethernet connection;    -   Set transmit power below the regions max;    -   Set which transmitters to accept connections with;    -   Allow enabling/disabling the IR Blaster (to prevent local        loopback); and    -   Restore factory defaults.

The front panel operation is also governed by firmware that controls thedisplay of an indication (by use of the 5 LEDs) that the system isbooting and/or fully booted.

As shown in FIG. 1, the system 10 provides an audiovisual product thatsupports point to multi-point transmission of High Definition (HD) videoand audio from the transmitter 11 to one or more remote receivers 12.The link between the transmitter and a receiver is either 100 mega-bitEthernet (wired) or 802.11a (wireless).

The audiovisual subsystem consists of two video JPEG2000 CODEC's(ADV202, one for Y and the other for C_(B)C_(R)), an audio CODEC, amulti-format video decoder and graphics digitizer (ADV7401), a HDMIreceiver (AD9398), a HDMI transmitter (AD9889) and a video encoder(ADV7321). A Xilinx XC3S500E FPGA is used to interface the AV subsystemto a 32 bit, 33 MHz PCI bus.

In addition to the HDMI and DVI video described, other and/or combinedembodiments can support other video standards such as Component Video,Composite Video and S Video.

This section describes the software architecture. The following sectionsdescribe the software in terms of the concurrent threads of execution,the calling hierarchy or layers of software, and the strategy forhandling incoming events in this event-driven system. Other aspects ofthe architecture are covered as well, such as error handling and thephysical layout of files and folders.

A. The Framework (XFw)

The XFw allows the software engineer to focus on the responsibilities ofa specific part of the system. The system can be broken down into ActiveObjects (AOs) that are responsible for a specific aspect of the systemas a whole. The framework provides and handles the common functionalityof AOs. Each AO runs in its own thread of execution and onlycommunicates with other AOs via events.

B. Layers

The framework is outside of the layering. Layering is a convenient wayto abstract the operating system and hardware to minimize changes in thesystem when either the hardware or operating system changes. Layeringalso facilitates testing of Active Objects and Logical Drivers. Therelationship between the framework and the layers is shown in FIG. 5.

In FIG. 5, each active object has its own thread of execution. AOsinterpret events and process them as prescribed by the design. AOs canpost events to logical drivers directly, but logical drivers mustPublish or Post events to AOs only through the framework. In general,AOs should never need to change due to change of OS or hardwareplatform.

The logical driver, shown in FIG. 5, hides the actual hardware/OSdesign. An AO posts events directly to a Logical Driver. A logicaldriver communicates with the system in one of two ways. It can publishan event through the framework or it can post an event directly to anactive object by using the framework. Logical drivers will have a threadthat blocks on tend to have a thread and therefore tend to be dependenton the operating system. They provide the highest level of abstractionwithin the system. The Logical Drivers should not need to change whenthe hardware changes, but probably will require change to accommodate achange in OS.

The redirector provides a convenient hardware abstraction point. Bychanging the redirector you can direct the requests to any hardwaredriver you want. For example: If a redirector is used to make calls toan RS-232 serial port (which is a character driver) it is easy to changethe redirector to call another character driver such as an I²C driver.Logical drivers or active objects may call a redirector. However, onlyone object (thread context) may call a redirector. Redirectors do nothave threads and merely provide functionality. A redirector mayimplement user mode driver functionality (such as MMIO) directly.Finally in

FIG. 56, the kernel modules interact with the hardware. Only redirectorscall kernel modules.

An example layer system is shown in FIG. 6.

The system architecture can be illustrated by breaking it down intodifferent functional areas and model them as classes. The overall classdiagrams for a transmitter and receiver are shown in FIGS. 7 and 8,respectively. There are different types of classes, namely:

1. Active Object Classes: An active object is a state machine with athread and event queues. It provides an event interface to otherobjects. It is responsible for modeling the behaviors of one functionalarea.

2. Passive Object Classes: A passive object is an object without athread. It provides a function interface to other objects. It does notexhibit state behaviors or process events. Examples include database andutility objects.

3. Logical Driver Classes: A logical driver is an object with a threadand an event queue. Like active objects, it provides an event interfaceto other objects. However it does not exhibit state behaviors. It servesas an interface to device drivers and network sockets. For example, itis used to convert interrupt status into events.

4. Redirector Classes: A redirector object provides an interface to adevice driver or kernel services in order to isolate the upper layersfrom the actual driver or kernel implementation. It allows us to onlyswap out the redirector when the underlying hardware or OS is changed.

For classes that are similar or the same for both transmitter 11 andreceivers 12, they are shared between the two architectures. As seen inFIGS. 7 and 8, most classes fall into this category. For classes thatdiffer significantly for transmitter 11 and receivers 12, they aredistinct in each architecture, such as AOAvSessionTx and AOAvSessionRx.

Active objects are shown as packages in the class diagrams. Each packageis composed of the active object itself, as well as any state machineobjects implementing the orthogonal regions. This is an overview of thefunctions of statecharts and class diagrams for active objects:

1. AOSysManager—This is the main active object responsible for overallsystem control and coordination among active objects. Its tasks include(1)

System initialization and reset, (2) Firmware upgrade, and (3)Management of wireless and wired network interface.

2. AOLocalUi—This is the active object that manages the Local UserInterface, which includes an LCD front panel and buttons on atransmitter, and signal strength LEDs on a receiver. It handles passingof IR data to the LPC (ARM slave processor). Its tasks include (1)Initialization and firmware upgrade of the ARM slave processor, (2)Button input detection, (3) LCD display and mode control, and (4) LEDcontrol.

3. AOAvControlTx—This active object controls the AV subsystem hardwareof a transmitter. Its tasks include (1) Hardware initialization, (2)Hot-plug assertion, (3) Video mode detection and configuration, (4) HDCPauthentication, and (5) Capturing HDMI control packets (e.g. SPD, ISCR1,ISCR2, ACP) from hardware.

4. AOAvControlRx—This active object controls the AV subsystem hardwareof a receiver. Its tasks include (1) Hardware initialization, (2)Hot-plug detection, (3) Video mode configuration, (4) HDCPauthentication, (5) Writing HDMI control packets (e.g. SPD, ISCR1,ISCR2, ACP) to hardware.

5. AOAvSessionTx—This active object manages the point-to-multipointlinks and sessions from the transmitter to receivers. Its tasks include(1) Link management, including (a) Discovery and validation of newreceivers, (b) Assignment of IP addresses to new receivers, and (c)Polling the status of existing receivers; and (2) Session management,including (a) Session establishment, e.g. sending RTP and A/V parametersto receivers, (b) Receiving audio/video frames from LDAV logical driverand multicast them to receivers using RTP via the LDRTPAudio/LDRTPVideological drivers, and (c) Exchanging CEC, HDCP and HDMI control packetswith receivers.

6. AOAvSessionRx—This active object manages the point-to-point link andsession from a receiver to the transmitter. Its tasks include (1) Linkmanagement, including (a) Probing for a transmitter, (b) Joining atransmitter, and (c) Responding to poll requests from the associatedtransmitter; and (2) Session management, including (a) Sessionestablishment, e.g. receiving RTP and A/V parameters from thetransmitter, (b) Reassembly of audio/video frames from received RTPpackets, (c) Time synchronization with the transmitter, (d) Playback ofaudio/video frames to LDAV logical driver, and (e) Exchanging CEC, HDCPand HDMI control packets with the transmitter.

7. AOCmdCtl—This active object manages the flow of command and controldata from a transmitter to a receiver, and vice versa. Its tasks include(1) reading serial data from the serial logical driver, (2) processinglocal IR data (from LDLocalUi), (3) processing remote IR data (from themaster receiver), (4) for transmitter, sending the IR & serial data tothe master receiver via AOAvSessionTx unless serial pass through isdisabled, (5) for receiver, sending the IR & serial data to thetransmitter via AOAvSessionRx, (6) writing received serial data to theserial logical driver, unless serial pass through is disabled, (7)publishing IR data to AOLocalUi, and (8) handling CEC.

The AO applications described herein are by way of example. Alternativescould be XFApp or other suitable applications.

The RTP stack protocol is used to transport the audio and videoinformation over the multicast. RTP is a known and popular protocol fortransporting real-time data such as audio and video over multicast orunicast network services. It typically runs on top of UDP/IP. It adds alight-weighted header (min 12 bytes) to each packet to carry thesequence number, timestamp, source identifier, etc. An accompanyingprotocol RTCP provides control functions such as QoS (Quality ofService) reporting and time synchronization.

At least two commercially available RTP stacks are suitable for use inthe present embodiments. They are:

ccrtp (http://www.gnu.org/software/ccrtp/)

jrtplib (http://research.edm.luc.ac.be/jori/jrtplib/jrtplib.html)

Although both are suitable, jrtplib is preferred.

Audio and/or video data can be encrypted as they are transmitted acrossthe network, wired or wireless. Various encryption programs areavailable, including standard cipher algorithms, Secure Socket Layer,OCF (OpenBSD Crytographic Framework), Secure RTP, IPSec, and Openswan.Other encryption schemes will undoubtedly arise in the future.Presently, Openswan is preferred. There are also several approaches forkey management, including fixed shared keys, manually set shared keys,and dynamic keys such as described athttp://www.securemulticast.org/msec-index.htm. Either manual or fixedshared keys are preferred.

FIG. 9 illustrates the transmission of a video frame using RTP at thetransmitter 11. As shown, the hardware signals an interrupt after it hastransferred a compressed video frame via DMA to system memory. It breaksthe frame into multiple data blocks, leaving gaps in-between forsoftware to insert headers. FIG. 9 shows block i−1 and block i asexamples. The device driver wakes up the logical driver which is pendingat select( ). The logical driver converts the interrupt status into aframe ready event for the active object. If it is in the right state,the active object passes the frame to the LDRTPVideo logical driver.LDRTPVideo requests the crypto driver to encrypt the frame via an ioctl(). When it returns, the encryption request has been queued up by thedriver. The crypto driver notifies the completion of encryption bywaking up LDRTPVideo asynchronously. For each data block in the frame,LDRTPVideo adds an RTP header by calling SendFrameInPlace( ). Sincespace for the header is already reserved, there is no need to allocate anew buffer and copy the data block. LDRTPVideo sends each RTP packet tothe socket by calling sendto( ). It copies the packet to a bufferallocated by the kernel IP stack. After sendto( ) is called for allpackets in a frame, LDRTPVideo sends an event to the active object tosignal that the frame has been sent (to the socket). The active objectreleases the ownership of the frame buffer back to the device driver sothat it can reuse it to store a new frame. The IP stack adds the UDP andIP headers and asks the network driver to transmit the packet. Thenetwork driver sets up a DMA transfer to send the packet to hardware.

FIG. 10 illustrates the reception of a video frame at the receiver 12.As shown, the LDRTPVideo logical driver waits for packets to arrive bycalling select( ) at the socket. When a packet arrives at the networkinterface, the packet is processed by the kernel IP stack. Since apacket is ready to read, select( ) returns. LDRTPVideo calls recvfrom( )to read the packet into its buffer. Packet data is copied to the bufferprovided by LDRTPVideo. LDRTPVideo call POPlaybackBuffer::StorePacket( )to store the received packet into the frame buffer to reassemble acomplete frame. For efficiency, event is not used here and a function iscalled directly. Note that the buffer memory is allocated by the AVsubsystem device driver. LDRTPVideo calls crypto driver to decrypt theframe when the frame is complete. The crypto driver notifies LDRTPVideoasynchronously when decryption is done. LDRTPVideo marks the frame asplayable. The AV subsystem FPGA issues an interrupt when it hascompleted a frame transfer to AD202 decoder. This interrupt requestssoftware to update the frame pointer in FPGA. Note that the framepointed to by the original frame pointer in FPGA may still be accesseduntil the next interrupt. The device driver wakes up the LDAVDatalogical driver. LDAVData converts the interrupt status into a datarequest event for the active object. The active object performs timesynchronization and gets the frame to playback from the frame buffer. Itpasses the frame pointer to LDAVData via an event. LDAVData passes theframe pointer to the device driver. The device driver sets up the videoframe pointer in FPGA as the next frame to transfer to AD202 decoder.

In order to achieve maximum transmission throughput it is preferable toavoid having the processor copy the data. Therefore a DMA controllerwithin the A/V subsystem FPGA transfers the video/audio data into framebuffers owned by the A/V subsystem driver. The FPGA has an array ofpointers to 16 video frame buffers and 16 audio frame buffers. The framepointer arrays will be initialized by the processor during startup. TheFPGA will iterate through all the frame buffers before beginning at thefirst one again. The frame buffers will be memory mapped by the A/Vsubsystem logical driver in order to avoid having to copy the data intouser space. The A/V subsystem driver will notify the logical driver ofthe arrival of data. If there is an active session the logical driverwill then send an event to AOAvSessionTx to transmit the data, usingRTP. Note that the marker bit in the RTP header will be set to indicatethe start of a frame. The payload type field will indicate whether thepacket is video or audio.

FIG. 11 shows example transmit frame buffers. To prevent IPfragmentation the maximum data payload should be the maximum UDP packetsize (1472) minus the RTP header, IP header and IPSec header size. Inaddition the DMA destination address must allow space for the RTP, IP,and IPSec headers to be inserted by the application. The FPGA allows themaximum data packet size and reserved space for a header to be set bythe processor.

FIG. 12 illustrates how the FPGA places data in a single video framebuffer. The first data packet of the frame has a header appended by theFPGA specifying the timestamp, frame number data packet count, order andsize of the chrominance and luminance components.

The receiver 12 audiovisual subsystem data interface includes anaudiovisual subsystem driver that owns an array of receive video andaudio frame buffers. An example receive frame buffer format is shown inFIG. 13. The receive frame buffers will be memory mapped by the A/Vsubsystem logical driver in order to avoid having to copy the data intouser space. Video and audio packets minus the RTP header are written toa frame buffer by the AOAvSessionRx object. When a complete frame hasarrived the FPGA is informed of the new frame and the next frame buffercan start to be filled by the processor. A frame complete interruptinforms the processor that the frame has been transferred to the decoderand is ready to receive the next frame.

The receivers 12 operate in synchronism. Continuous audio and videostreams are delivered in real-time. When using asynchronous networks fordata transmission, however, timing information of the media unitsproduced gets lost and a mechanism is required to ensure continuous andsynchronous playback at the receiver side. Inter-stream synchronizationbetween audio and video streams, as well as between different receiversare also required.

The paper, Laoutaris, “A Survey of Playout Schedulers” presented anumber of synchronization schemes. For this embodiment, thesynchronization scheme is a time-orientated one and uses an approximatedclock. Media units are time-stamped at the transmitter and the receiverclock is synchronized to the transmitter clock at regular intervals. InRTP, this is achieved by using the timestamp field in RTP headers andsending the transmitter clock regularly via RTCP Sender Report packets.

Packets arriving at the receiver are buffered in order to compensate forvarying propagation times between the transmitter and receiver. Thejitter buffer should be sized to allow for the largest network delay.The total end-to-end delay of a Media unit is fixed and is composed of avariable network delay and buffering delay introduced by the jitterbuffer.

Packets that arrive at the receiver with a timestamp larger that thelocal clock are buffered. Packets that arrive with timestamps smallerthe local clock are discarded. Packets are extracted from the buffer andplayed when the local clock equals their timestamp.

The following sections present the formal and concrete design of thesynchronization scheme introduced above. They discuss intra-stream andinter-stream synchronization, as well as how to incorporate reliabilityinto RTP.

1. Intra-Stream Synchronization (Between Transmitter and Receivers)

In this scheme, there is no concept of a global clock. Rather, receiversin the network approximated the transmitter clock and use it to derivethe playback clock (or virtual clock).

A. Normalized Offset Between Transmitter and Receiver Clocks

The clocks of the transmitter and receivers are 32-bit countersincremented periodically by the encoding and decoding hardwarerespectively. They are not physically synchronized to one another sothere are offset and will drift among them.

Referring to FIG. 14, T_(tx)(t) and T_(rx)(t) are the step functionsrepresenting the transmitter and receiver clock values at time trespectively. The goal of synchronization is to produce an accurateestimate of T_(tx)(t), denoted by T_(tx, estimated)(t), at the receiver.To achieve this, the transmitter periodically sends its clock value tothe receiver via the “RTP timestamp” field in the Sender Report packets.For example at some time t_(n), it sends T_(tx)(t_(n)) to the receiver.

When the receiver gets the Sender Report packet at time t_(m), itmeasures the current offset between the transmitter and receiver clocks,T_(offset, measured)(t_(m)), by calculating the difference betweenT_(tx)(t_(n)) carried by the Sender Report and the current receiverclock value, T_(rx)(t_(m)). That is:

T _(offset,measured)(t _(m))=T _(tx)(t _(n))−T _(rx)(t _(m))

Note that all clock and offset values are treated as 32-bit signedintegers using 2's complement. That is, 0x7FFFFFFF is the most positiveclock value. After one increment, it becomes 0x80000000 which isinterpreted as the most negative clock value.

When the offset is positive, we say the transmitter clock is leading thereceiver clock. When negative, the transmitter clock is lagging thereceiver clock. When zero, the two clocks are in phase. Discontinuityhappens when the phase difference between the two clocks crosses 180°.The offset jumps from the most positive to the most negative (or viceversa). For example, consider c=a−b. If a is 0x00001000 and b is0x80001000, a−b=0x80000000 (most −ve). When b is incremented by just oneto be 0x80001001, a−b=0x7FFFFFFF (most +ve). This would cause problemswhen calculating averages using signed arithmetic. For example, theaverage of 0x80000000 and 0x7FFFFFFF is zero which incorrectly meansin-phase.

To avoid the above problem, we ensure the offset does not overflow orunderflow. Let the initial offset at t_(o) beT_(offset, measured)(t_(o)), we have:

T _(offset,measured)(t _(o))=T _(tx)(t _(p))−T _(rx)(t _(o)),

where t_(p) is the time when the Sender Report is sent. We derive thenormalized receiver clock (FIG. 15) and the normalized offset measuredat some time t_(m) (FIG. 16) as:

T′ _(rx)(t)=T _(rx)(t)+T _(offset,measured)(t _(o))

T′ _(offset,measured)(t _(m))=T _(tx)(t _(n))−T′ _(rx)(t _(m))  (1)

To verify, at initial time t_(o),

T_(offset, measured)^(′)(t_(o)) = T_(tx)(t_(p)) − T_(rx)^(′)(t_(o)) = T_(tx)(t_(p)) − T_(rx)(t_(o)) − T_(offset, measured)(t_(o)) = T_(offset, measured)(t_(o)) − T_(offset, measured)(t_(o)) = 0

Now the initial offset is normalized to 0. Over time, drift between thetransmitter and receiver clocks will cause the offset to change slowly.Since the rate of change is so slow, it is safe to assume that theoffset will not overflow or underflow before synchronization restarts(for a new RTP session). To validate this argument, assume theresolution of the transmitter and receiver clock is 1 ms (1 incrementper ms) and the clock drift is 1 s per minute (which is huge). It wouldtake 4 years for the offset to overflow or underflow.

Using values of T′_(offset, measured)(t_(m)) for various t_(m), we canconstruct the step function T′_(offset, measured)(t) representing the(normalized) measured offset between the transmitter and receiver clocksat time t (FIG. 16).

B. Estimation of Transmitter Clock by Receivers

In equation (1), because of network and processing delays, t_(n) andt_(m) are not identical and hence the measured offset differs from theactual one defined as FIG. 15:

T′ _(offset,actual)(t)=T _(tx)(t)−T′ _(rx)(t)

We can represent the measurement error as a constant error ε caused byfixed delays, plus a varying component δ(t) caused by jitter. Now wehave the relation:

T′ _(offset,actual)(t)=T′ _(offset,measured)(t)+ε+δ(t)  (2)

Because of clock drift between the transmitter and receiver,T′_(offset, actual)(t) is not constant, but changes slowly over time.However within a small time window, it is almost constant and can betreated as so. Also, δ(t) averages to zero. Using these two properties,we can estimate the actual offset by calculating the running-average asfollows:

Let average_(N,L,S)(f(t)) be the running average of f(t) over the last Nsamples of f(t), with the largest L samples and the smallest S samplesignored to avoid errors caused by extreme jitters. We have:

$\begin{matrix}{{T_{{offset},\; {estimated}}^{\prime}(t)} = {{{average}_{N,L,S}\left( {T_{{offset},\; {measured}}^{\prime}(t)} \right)}\mspace{166mu} = {{{average}_{N,L,S}\left( {{T_{{offset},\; {actual}}^{\prime}(t)} - ɛ - {\delta (t)}} \right)}\mspace{166mu} \approx {{{average}_{N,L,S}\left( {T_{{offset},\; {actual}}^{\prime}(t)} \right)} - ɛ}\mspace{166mu} \approx {{T_{{offset},\; {actual}}^{\prime}(t)} - ɛ}}}} & (3)\end{matrix}$

The values of N, L and S are to be determined empirically. Later weprove that the constant error ε can be cancelled out.

Now we introduce T′_(offset, used)(t) to be the offset function actuallyused by the receiver. In the simplest case, we use the estimated offsetfunction directly:

T′ _(offset,used)(t)=T′ _(offset,estimated)(t)  (4)

However, as we shall see later, changes in T′_(offset, estimated)(t)over time (due to clock drift) may cause the playback clock to cross theframe boundary which would result in frame skip/repeat. In terms of userexperience, it is arguable whether it is better to have frequent butsmall skip/repeat, or to have rare but large skip/repeat. IntroducingT_(offset, used)(t) gives us the flexibility to update it withT_(offset, estimated)(t) only when their difference is larger than acertain threshold. For simplicity, we assume (4) holds for the rest ofthe discussion.

With T_(offset, used)(t) defined, the receiver estimates the transmitterclock as below (FIG. 17). FIG. 18 demonstrates how the estimatedtransmitter clock follows the actual one.

$\begin{matrix}{{T_{{tx},\; {estimated}}(t)} = {{{T_{rx}^{\prime}(t)} + {T_{{offset},\; {used}}^{\prime}(t)}}\mspace{135mu} = {{T_{rx}^{\prime}(t)} + {T_{{offset},\; {estimated}}^{\prime}(t)}}}} & (5)\end{matrix}$

C. Playback Clocks

Once the receiver has an estimate of the transmitter clock,T_(tx, estimated)(t), it can derive the playback clocks from it. Becauseof the timing difference between audio and video decoding, there areseparate playback clocks, T_(playback, audio)(t) andT_(playback, video)(t). They are derived from the estimated transmitterclock as follows:

T _(playback,audio)(t)=T _(tx,estimated)(t)−T _(delay,audio)  (6a)

T _(playback,video)(t)=T _(tx,estimated)(t)−T _(delay,video)  (6b)

where T_(delay,audio) and T_(delay,video) are constant non-negativeplayback delays (in transmitter clock unit) for audio and videorespectively. They allow the receiver to buffer up packets to absorbnetwork and processing jitters.

Now we prove that the playback clock is synchronized to the originaltransmitter clock within a constant delay. We take audio as example, butit can be generalized to video as well.

$\begin{matrix}{{T_{{playback},\; {audio}}(t)}\begin{matrix}{= {{T_{{tx},\; {estimated}}(t)} - {T_{{delay},{audio}}\mspace{14mu} {{from}\left( {6a} \right)}}}} \\{= {{T_{rx}^{\prime}(t)} + {T_{{offset},\; {estimated}}^{\prime}(t)} - {T_{{delay},{audio}}\mspace{14mu} {{from}(5)}}}} \\{\approx {{T_{rx}^{\prime}(t)} + {T_{{offset},\; {estimated}}^{\prime}(t)} - ɛ - {T_{{delay},\; {audio}}\mspace{14mu} {{from}(3)}}}} \\{= {{T_{tx}(t)} - ɛ - {T_{{delay},\; {audio}}\mspace{14mu} {by}\mspace{14mu} {definition}}}} \\{= {{T_{tx}(t)} - T_{{delay},\; {audio}}^{\prime}}}\end{matrix}} & (7)\end{matrix}$

where T′_(delay,audio) is a constant equal to T_(delay,audio)+ε.

D. RTP Timestamps

In each RTP packet, the transmitter puts the transmitter clock value atsampling instant t_(s), T_(tx)(t_(s)), into the “timestamp” field of theRTP header. This indicates to the receiver when this packet should beplayed according to the playback clock, for both audio and videopackets. In case time-stamping takes place after video compression, thedelay introduced by compression should be compensated to ensure that thepacket timestamp represents the sampling instant.

On the receiver side, it saves the received RTP packets into the jitterbuffers. The decoding hardware interrupts the processor at fixedfrequency equal to the frame rate (according to the receiver clock). Wecall it the frame interrupt. When handling the frame interrupt, thereceiver compares the current playback clocks, T_(playback,audio)(t_(i))and T_(playback,video)(t_(i)) to the “timestamp” fields of the receivedRTP packets to determine which audio and video packets are to bedecoded.

Taking audio as example, the receiver checks whetherT_(playback,audio)(t_(i)) falls into the playback period of eachplayable audio frame in the jitter buffer, starting from the oldest. Asthe phrase is used herein, a frame is playable if all of its packetshave been received and the frame is decrypted. An audio frame is a setof consecutive audio packets with the same timestamp. The playbackperiod of a frame is defined as the range:

1. if a next playable frame is available in jitter buffer,

-   -   [timestamp of this frame, timestamp of next good frame)

2. otherwise,

-   -   [timestamp of this frame, ∞)        Denote the playback period of a frame as [P_(lower bound),        P_(upper bound)). There are three possibilities:

1. T_(playback,audio)(t_(i)) within the range.

-   -   (i.e. T_(playback,audio)(t_(i))−P_(lower bound)>=0 and        P_(upper bound)−T_(playback,audio)(t_(i))>0)    -   The receiver plays this audio frame.

2. T_(playback,audio)(t_(i)) earlier than the range.

-   -   (i.e. P_(lower bound)−T_(playback,audio)(t_(i))>0)    -   This means the receiver clock has been running faster than the        transmitter clock and it is not the time to play this audio        frame yet. The receiver either plays silence, white noise, or        the previous frame if one is available (depending on which        scheme is best in concealing error.)

3. T_(playback,audio)(t_(i)) later than the range.

-   -   (i.e. T_(playback,audio)(t_(i))−P_(upper bound)>=0)    -   This means the receiver clock has been running slower than the        transmitter clock and the time to play this audio frame has        passed. The receiver skips this audio frame and repeats the        check on the next frame in the buffer.

As stated before, clock values are interrupted as 32-bit signedintegers. This automatically handles the wrap-around cases duringcomparison. Discontinuity in computing differences is not an issue heresince the playback clock is very close the timestamp (far from being180° out of phase).

In the above example, we observe that packets are sent according to thetransmitter clock T_(tx)(t) and they leave the jitter buffer accordingto the playback clock T_(playback,audio)(t). Since the playback clock issynchronized to the transmitter clock within a constant delayT′_(delay,audio), the number of packets in transit and in the jitterbuffer is equal to the number of packets sent in the durationT′_(delay,audio), which is a constant. As the network and processingdelays vary, the number of packets in the jitter buffer varies. Provideda large enough buffer, buffer underflow should not happen. In thisdesign, the size of the jitter buffer is determined empirically withoutproof.

2. Inter-Stream Synchronization

A. Between Audio and Video Streams

Using intra-stream synchronization explained in the previous section, wecan synchronize the playback clock of a stream at the receiver to thetransmitter clock. As in this design both audio and video streams aretime-stamped using the same clock source, inter-stream synchronizationis implicitly achieved by virtue of intra-stream synchronization.

For video stream, all packets of a video frame share the same timestampas the first packet of the frame. For audio stream, all packets sampledin the duration of a video frame have the same timestamp as the firstvideo packet of the frame. We call the set of those audio packets havingthe same timestamp to be in an audio frame.

As the decoding time for video packets is longer than that for audiopackets, in order to ensure that video and audio packets with the sametimestamps output at the decoder simultaneously, the video playbackclock T_(playback,video)(t) should be ahead of the audio playback clockT_(playback,audio)(t). That is, the audio playback delay should belarger than the video playback delay and we have:

T _(delay,audio) =T _(delay,video)+η  (8)

where η is the absolute value of the difference between video and audiodecoding times (in transmitter clock unit). η is to be determinedempirically and has been suggested to be around 2 video frame periods.

B. Among Receivers

Using intra-stream synchronization, the playback clock of a receiver issynchronized to the transmitter clock within a constant delay. Sincethere is only one transmitter in a network, if all receivers choose thesame playback delay (T_(delay,audio)/T_(delay,video)), they areeffectively synchronized to each other.

As this synchronization scheme is software-based, the underlyinghardware clocks of receivers are still not synchronized. For example,frame interrupts may happen at different times on different receivers.Assuming ideal software synchronization (i.e.T_(tx,estimated)(t)=T_(tx)(t)), there are still errors caused by phasedifferences among hardware clocks, which is upper-bounded by the frameperiod as illustrated in FIG. 19.

At 30 frames per second, the error is limited by 33 ms. This isacceptable since a delay less than 100 ms will be perceived asreverberation rather than echo.

3. Reliable Multicast

RTP runs on top of UDP which is an unreliable transport protocol. RTPitself does not provide reliability services. In general, this isacceptable to multimedia streams since the emphasis is on efficiency andon meeting the timing requirements of the majority of packets. Aretransmitted but delayed packet would be of little use to the user.

While the loss of a single video frame may not be perceived by the user,the loss of an audio frame may be more noticeable. In order to provide ahigh-quality service, we extend RTP to support reliable multicast foraudio. The basic theory has been brought up in an RFC draft called “RTPextension for Scalable Reliable Multicast” dated 1996, which we improveupon and modify into the present environment.

A. Sequence Numbers

The order of packets is identified by a 16-bit sequence number in theRTP header. It increments by one for each packet sent. When it reaches0xFFFF, it wraps around to 0. Sequence numbers are interpreted as 16-bitsigned shorts using 2's complements. Signed arithmetic automaticallyhandles wrap around cases when comparing sequence numbers. Given twosequence numbers a and b, we say a leads (is ahead of) b if a−b>0 and alags (is behind) b if a−b<0.

A tricky situation occurs when a and b are offset by about half of thetotal range. A small change in a or b would cause a−b to jump from themost positive to the most negative. In other words, it is ambiguouswhether a leads b by vice versa. In reality, we will only comparesequence numbers within a small window relative to the total range, andtherefore the ambiguity would not occur.

To assist frame re-assembly, the fixed RTP header is extended to includefields “frame number” and “packet number”. Frame number increments byone for each audio/video frame sent and packet number identifies theposition of the packet in the frame. They help locate the space in theframe buffer to store a received packet. Like sequence number, the framenumber is interpreted as a signed integer (32-bit).

B. Detection of Lost Packets

Unlike TCP in which senders detect packet losses via positiveacknowledgements (ACKs), this design places that responsibility toreceivers by using negative acknowledgements (NACK), which reducescommunication overheads.

Let the sequence number of the first received packet be N₀. When thereceiver gets the first packet, it sets the expected sequence numberN_(expected) to N₀+1.

When the receiver gets a packet, it compares its sequence number N_(k)to N_(expected). There are several possibilities:

1. N_(k)=N_(expected)

-   -   This is the normal case in which the sequence number of the        received packet matches the expected one. It means it is an        in-order packet. N_(expected) increments by one.

2. N_(k) leads N_(expected)

-   -   (N_(k)−N_(expected)>0)    -   This indicates some packet(s) are missing, which may be caused        by packet loss or out-of-order delivery. The number of missing        packets is equal to N_(k)−N_(expected). We add entries for all        missing packets to a linked list of MissingPacket objects        defined as:

class MissingPacket   {  short seqNum;  // sequence number of packet int nackTime;  // time to send NACKs to transmitter  int ignoreTime; //until which to ignore duplicate NACKs   }

-   -   This linked list stores the sequence numbers of missing packets.        The purpose of nackTime is to avoid all receivers missing the        same packet from sending NACKs at the same time. The purpose of        ignoreTime is to allow the receiver to ignore duplicate NACKs        arriving within a short period. Their uses are explained in the        next section.        -   Finally, we set N_(expected)=N_(k)+1.

3. N_(k) lags N_(expected)

-   -   (N_(k)−N_(expected)<0)    -   This indicates an earlier missing packet or a duplicate packet        has arrived. If it is a duplicate, it is discarded. If it is a        missing packet, it is either delivered out-of-order or        retransmitted. In either case, it is stored to the frame buffer        and its entry in the linked list of MissingPacket objects is        removed.

C. NACK Suppression

In a multicast environment, a packet is sent to multiple receivers. Ifone receiver misses a packet, it is likely that others miss it too. Ifall receivers send NACKs at the same time, it may cause networkcongestion and result in more packet losses. Besides, since a successfulretransmission after a single NACK is seen by all receivers, multipleNACKs are unnecessary.

The solution is to have receivers waiting for random times beforesending NACKs. We maintain the timer by MissingPacket::nackTime (callednackTime for brevity). It is more scalable than creating aframework-based timer object for each missing packet. When a missingpacket is detected, we initialize nackTime according to:

nackTime=T _(rx)(t _(c))+multiplier(timeout_count)*T _(nack) _(—)_(wait)  (9)

where T_(rx)(t_(c)) is the current receiver clock value, timeout_countis the number of times the timer expired (which is zero here) andT_(nack) _(—) _(wait) is the initial duration to wait before sendingNACKs. The function multiplier(n) determines by how much the timeoutperiod is increased after each time-out. The function is to bedetermined but it is required that multiplier(0)=1 andmultiplier(n+1)>=multiplier(n). Possibilities include linear (1+n),exponential (2̂n) or constant (1). Incidentally, ignoreTime isinitialized to T_(rx)(t_(c)) such that NACKs will not be ignoredinitially (see later). The choice of T_(nack) _(—) _(wait) will bediscussed later.

The receiver checks for time-outs periodically by means of interrupts,such as frame interrupts. It scans the list of MissingPacket objects andfor each object compares nackTime against the current receiver clockvalue T_(rx)(t_(c)). If the timer expires (i.e.T_(rx)(t_(c))−nackTime>=0), it multicasts NACKs to request forretransmission. Note that multiple sequence numbers can be carried byone NACKs packet to reduce overhead. After sending NACKs, the receiverincrements timeout_count and resets nackTime according to (9) to waitfor retransmission. It also sets up ignoreTime as explained in the nextparagraph.

If the missing packet is received before time-out, the timer is canceledand its entry in the linked list of MissingPacket objects is removed. Ifa NACKs packet is received and is not ignored (see later), the receivertreats it as time-out. It increments timeout_count and resets nackTimeaccording to (9) to wait for retransmission. To avoid duplicate NACKscausing nackTime to increase multiple times in a short period, afterresetting nackTime, the receiver sets ignoreTime halfway between thecurrent time and the new nackTime. If a NACKs is received beforenackTime (i.e. T_(rx)(t_(c))−ignoreTime<0), it is ignored. As a specialcase, set ignoreTime to the current time to not ignore any NACKs.

The original RFC draft requires each receiver sets its initial wait time(T_(nack) _(—) _(wait)) to be a random number within in a certain range[C1, C2] where C1 and C2 are constants. The purpose is to avoidreceivers sending NACKs simultaneously. In this design, time-outs arepolled by means of interrupts. Assuming frame interrupts are used, theresolution of timeout is limited by the period of frame interrupts,which is 16 ms at 30 frames/sec. With such course resolution, in orderto provide enough randomness, the range [C1, C2] should be large. Itmeans a longer wait time before sending NACKs and hence calls for alonger playback delay, which is undesirable.

Fortunately, by the fact that the hardware clock of different receiversare not synchronized to each other, there is a random phase differencebetween the frame interrupts on any two receivers. As a result, there israndomness in the time when a receiver checks for time-outs and sendsNACKs. Therefore, we can choose [C1, C2] to be a small range, yetprovides enough randomness. C1 and C2 are to be determined empirically.Possible values are C1=frame period (in receiver clock unit) andC2=2*C1.

D. Retransmission Suppression

Even with NACKs suppression, multiple NACKs from different receivers forthe same missing packet may still reach the transmitter within a shortperiod. It is unnecessary to retransmit the packet multiple times. Thesolution is to start a timer after retransmitting a packet. If NACKs forthe same packet arrive before time-out, they are ignored.

First we introduce the class ReTxPacket:

class ReTxPacket   {  short seqNum;  // sequence number of packet  intignoreTime; // until which to ignore duplicate NACKs   };

After a packet is re-sent, the transmitter adds an entry for it in thelinked list of ReTxPacket objects. The entry contains its sequencenumber and the time until which retransmission requests for the samepacket are to be ignored. The time is initialized to:

ignoreTime=T _(tx)(t _(c))+T _(ignore)  (10)

where T_(tx)(t_(c)) is the current transmitter clock value andT_(ignore) is the ignoring duration. T_(ignore) is a constant to bedetermined empirically.

When a NACKs is received, the transmitter checks the sequence number(s)it contains against those in the linked list of ReTxPacket objects. Foreach matched object, it checks if ignoreTime has expired. If not (i.e.T_(rx)(t_(c))−ignoreTime<0), the retransmission request for that packetis ignored.

The transmitter loops through the linked list of ReTxPacket objectsperiodically to purge entries with expired ignoreTime.

Next, the transmitter A/V subsystem control is described with respect toFIG. 20. The HDMI receiver supports the following input pixel encodings.

4:4:4 YCrCb 8 bit

4:2:2 YCrCb 8, 10, and 12 bit

RGB 8 bit

The output of the HDMI receiver is connected to the digital interface ofa multi-format video decoder and graphics digitizer. Regardless of theinput pixel encoding the HDMI receiver colorspace converter must be usedto set the pixel encoding to 4:4:4 YCbCr 24 bit as this is required bythe Component Processor of the video decoder. The colorspace converterof the video decoder is used to convert the output to 4:2:2 YCrCb 16 bitnecessary for the JPEG2000 encoder. ally to purge entries with expiredignoreTime.

HDMI carries auxiliary data that describe the active audio and videostreams. This includes the following data.

Auxiliary Video Information (AVI) InfoFrame

Audio InfoFrame

Source Product Description (SPD) InfoFrame

Audio Content Protection (ACP) Packets

International Standard Recording Code ISRC1/ISRC2 Packets

The auxiliary data needs to be sent from the source to the sink. In ourproduct this data is treated as out-of-band information and will be sentas control packets over the wired or wireless link. The format ofinfoFrames and infoPackets can be found in the CEA-861B specification.

When the video source is DVI separate inputs are used for audio. Anaudio CODEC is used to generate an I2S digital audio stream.

The host processor is required to perform initial configuration of theA/V Subsystem. In addition configuration is required whenever the videoresolution or audio format of the source changes. At a high level thefollowing is required.

The HDMI Receiver of FIG. 20 performs the following:

-   -   1. Set audio PLL and VCO range    -   2. Set HSYNC and VSYNC source, polarity and timing    -   3. If the input encoding is not 4:4:4 YCrCb enable the color        space converter and program the coefficients    -   4. Enable BT656—Start of Active Video (SAV) and End of Active        Video (EAV) controls    -   5. Enable SPDIF audio output    -   6. Monitor the New Data Flags (NDFs) to detect changes in the        auxiliary data    -   7. Read the AVI, audio and SPD Infoframes along with ACP and        ISRC1/C2 packets, to send to the receiver

The Video Decoder of FIG. 20 performs the following:

1. Set the global registers

2. Set the primary mode to HDMI support

3. Set video standard

4. Set the color space converter to output 4:2:2 YCrCb

5. Set the Component Processor registers

The JPEG2000 Encoder of FIG. 20 performs the following:

1. Set PLL registers

2. Set bus mode

3. Load encode firmware

4. Set encode parameters

5. Set dimension registers (If custom mode)

6. Start program

In order to support custom formats the dimension registers must be setusing the information available in the AVI InfoFrame.

The Audio CODEC (TLV320AIC33) of FIG. 20 performs the following:

1. Set the PLL

2. Select the input source

3. Setup the Programmable Gain Amplifier (PGA)

4. Select I2S output

5. Setup the DAC and outputs to provide local audio

The receiver A/V subsystem control will now be described with respect toFIG. 21.

In the receiver the host processor sends video frames to the JPEG2000decoder via the A/V subsystem FPGA. A HDMI transmitter receivesuncompressed video from the decoder and outputs an HDMI stream. Audioframes are sent to the A/V subsystem FPGA which after processingforwards the audio data to the HDMI transmitter as well as the audioCODEC. When the source is DVI audio is supplied via separate audioconnectors from the audio CODEC.

The host processor performs the following configurations:

JPEG2000 Decoder (ADV202)

-   -   Set PLL registers    -   Set busmode    -   Load decode firmware    -   Set decode parameters    -   Set dimension registers    -   Start program    -   For custom formats the values for programming the dimension        registers come form the AVI InfoFrame sent by the transmitter at        the start of a session and whenever a resolution change is        detected.

HDMI Transmitter (ADV9889)

-   -   Set audio type to S/PDIF    -   Set audio registers (N and CTS parameters)    -   Set input pixel encoding to 4:2:2 YCrCb 16 bit with embedded        syncs    -   Set color space converter to set the output pixel encoding to be        the same as the video source

Audio CODEC (TLV320AIC33)

-   -   Set the PLL    -   Set the volume control and effects    -   Select the analog output

The Display Data Channel will now be described.

The enhanced display data channel (E-DDC) is used by the Source to readthe Sink's Enhanced Extended Display Identification Data (E-EDID) inorder to discover the Sink's configuration and/or capabilities. HDMISources are expected to read the Sink's E-EDID and to deliver only theaudio and video formats that are supported by the Sink. All Sinkscontain an EIA/CEA-861B compliant E-EDID data structure accessiblethrough the E-DDC.

Extended EDID (E-EDID) supports up to 256 Segments. A segment is a 256byte segment of EDID containing one or two EDID blocks. A normal HDMIsystem will have only two EDID blocks and so will only use segment 0.The first EDID block is always a base EDID version 3 structure 128 bytesin length. This structure contains a Vendor Specific data block definedfor HDMI systems and holds the 2-byte Source Physical Address field usedfor CEC message addressing. The second EDID block is not used by HDMIdevices.

The HDMI transmitter reads EDID segment 0 of the connected displaydevice when the Hot-Plug-Detect is asserted and generates an EDID Readyinterrupt. The System processor can read the EDID segment via the I2Cbus and send it via an out-of-band packet to the transmitter.

The system acts as a Repeater with a Duplicator function i.e.Single-input, multiple-output device, where more than one output isactive. The transmitter needs to determine the video standard and audioformat to use based on the EDID data from all the receivers in thesystem. The video standard used must be suitable for the lowestresolution display.

The HDMI/DVI source also prevents all protected audiovisual data frombeing copied. Content protection is provided by High-bandwidth DigitalContent Protection (HDCP) specification version 1.10.

The HDCP Authentication protocol is an exchange between an HDCPTransmitter and an HDCP Receiver that affirms to the HDCP Transmitterthat the HDCP Receiver is authorized to receive HDCP Content. Thisaffirmation is in the form of the HDCP Receiver demonstrating knowledgeof a set of secret device keys. Each HDCP Device is provided with aunique set of secret device keys, referred to as the Device PrivateKeys, from the Digital Content Protection LLC. The communicationexchange, which allows for the receiver to demonstrate knowledge of suchsecret device keys, also provides for both HDCP Devices to generate ashared secret value that cannot be determined by eavesdroppers on thisexchange. By having this shared secret formation melded into thedemonstration of authorization, the shared secret can then be used as asymmetric key to encrypt HDCP Content intended only for the AuthorizedDevice. Thus, a communication path is established between the HDCPTransmitter and HDCP Receiver that only Authorized Devices can access.

Through a process defined in the HDCP Adopter's License, the DigitalContent Protection LLC may determine that a set of Device Private Keyshas been compromised. If so, it places the corresponding KSV on arevocation list that the HDCP Transmitter checks during authentication.Revocation lists are provided as part of the source media (i.e. on theDVD). Other authorized HDCP receivers are not affected by thisrevocation because they have different sets of Device Private Keys.

An HDMI Transmitter at the source (i.e. a DVD player) can initiateauthentication at any time. The HDMI Receiver responds by sending aresponse message containing the receivers Key Selection Vector (KSV).The HDCP Transmitter verifies that the HDCP Receiver's KSV has not beenrevoked.

The Receiver must gather the authentication data of all downstream sinksand report it back to the Transmitter. The required data is as follows.

-   -   KSV lists—The KSV's from all attached displays.    -   Device Count—The number of displays/repeaters in the connection        topology.    -   Depth—The number of connection layers in the topology.    -   Max_Cascade_Exceeded—A flag that is set if Depth exceeds a        maximum value (for example, 7).    -   Max_Devs_Exceeded—A flag that is set if the Device Count exceeds        a maximum value (for example, 127).

The HDMI transmitter EDID and HDCP controller is a state machine that isimplemented in hardware. Its purpose is to retrieve the EDID and KeySelection Vectors from downstream receivers. The following stepsillustrate the sequence in which the state machine performs the EDID andHDCP handling. This process takes place every time a Hot Plug Detect issensed (as described in step 1). It will also take place every time thetransmitter requests a re-authorization. In this case, software wouldbegin the re-authorization at step 4.

-   -   1. Hot Plug Detect goes high (toggled by the attached display)    -   2. The AD9889 automatically reads EDID segment 0. The EDID Ready        flag (0xC5[4]) is set once the EDID has been read successfully        and sends an EDID Ready interrupt to the system.    -   3. After receiving EDID Ready interrupt, the system software        evaluates the EDID. (EDID data is stored under at I2C address        0x7E, beginning at offset 0.    -   4. Once the Receiver has set the video/audio mode it then sets        the HDCP Desired bit (0xAF[7]) to high. The HDMI/DVI bit        (0xAF[1]) should not be changed after setting the HDCP Desired        bit.    -   5. The receiver's BKSV is reported in the BKSV registers        (0xBF-0xC3), the BKSV flag is set (0xC7[7]) (generates an        interrupt) and the BKSV count (0xC7[6:0]) is set to 0 by the        AD9889.    -   6. Upon receiving the BKSV flag interrupt, the system software        reads the BKSV and clears the BKSV Flag interrupt.    -   7. Once the BKSV flag is cleared, the AD9889 begins HDCP        encryption and checks if the receiver is a repeater.        -   a. If not a repeater, then HDCP initialization is complete            and the AD9889 begins HDCP management, the firmware will            know if this state is reached when HDCP Controller State            (0xC8) reaches state ‘4’. Process complete.            -   i. One BKSV should be stored by software at this point.            -   ii. DEVICE_COUNT=1            -   iii. DEPTH=0            -   iv. MAX_DEVS_EXCEEDED=0            -   v. MAX_CASCADE_EXCEEDED=0        -   b. If the Receiver is a repeater the AD9889 must complete            the HDCP repeater authentication, the HDCP Controller State            will not reach state ‘4’. Continue to step 8.    -   8. The AD9889 reads up to 13 KSV's from the downstream repeater        (the AD9889 can only process 13 at a time)    -   9. The AD9889 signals a BKSV Flag interrupt with the BKSV count        (can be up to 13)    -   10. System software reads bStatus from EDID memory space (device        0x7E) registers 0xF9 (LSB) and 0xFA (MSB). (Only read the first        time through this loop)    -   11. bStatus[6:0] contains DEVICE_COUNT.    -   12. bStatus[7] contains MAX_DEVS_EXCEEDED.    -   13. bStatus[10:8] contains DEPTH.    -   14. bStatus[11] contains MAX_CASCADE_EXCEEDED.    -   15. If either of the ‘MAX . . . ’ values are set to 1, then we        can exit this routine and just forward those status flags        upstream. (Depth and Device_Count don't matter when the maximums        are exceeded). Process Complete.    -   16. System firmware reads the BKSV Count to see how many valid        BKSV's are in EDID memory.    -   17. System firmware reads the BKSV's from the EDID memory. This        list of BKSVs is stored in a separate memory space inside the        AD9889 that can be accessed by using the I2C device address 0x7E        (instead of 0x72 or 0x7A). When you read from this I2C device,        up to 13 BKSVs from downstream devices will be in the memory        starting with offset 0x00 then 0x05 . . . etc.    -   18. System software clears the BKSV Flag interrupt.    -   19. If more KSV's remain (there are more than 13 downstream        devices) then go back to step 9. Software will know if more keys        remain from the DEVICE_COUNT field read in step 11.    -   20. System software now has a full list of all downstream        KSV's+the directly attached BKSV.

Once the Authentication is complete the AD9889 will manage the ongoingHDCP link authentication every 128 frames. A failure authentication willgenerate an HDCP/EDID Controller Error interrupt and restart the HDCPauthentication.

The system firmware should periodically check the state of the“Encryption ON” status bit (0xB8 [6]) while sending protected audio orvideo to ensure that HDCP is enabled. This should be checked with afrequency of no less than once every two seconds. Checking this bitprotects against third party meddling with the AD9889's registersettings to defeat HDCP.

The Transmitter must consolidate all downstream Receiver KSV lists intoa single list. The list is represented by a contiguous set of bytes,with each KSV occupying 5 bytes stored in little-endian order. The totallength of the KSV list is 5 bytes time the total number of downstreamsinks.

The Transmitter must also compare all DEPTH parameters from eachattached Receiver. The maximum reported DEPTH will be incremented by 1and reported to the source. If the new DEPTH is greater than 7, thenMAX_CASCADE_EXCEEDED shall be set to 1.

The Transmitter must also collect all DEVICE_COUNT parameters from eachReceiver, these numbers will be added together for a total DEVICE_COUNTto be reported to the source. If the total is greater than a maximumvalue (for example, 127), then MAX_DEVS_EXCEEDED shall be set to 1.

The authentication data must be forwarded to the source by the HDMIReceiver in the Transmitter. An issue here is that AD9398 doesn'tprovide documented registers for doing this. This issue will be solvedwith a new ADI HDMI Receiver (AD9399) that will be used in theproduction hardware.

In addition to HDCP a source may use the ACP packet to conveycontent-related information regarding the active audio stream. ACPpackets received from the source need to be sent to all receivers in thesystem. The content of the ACP packet is used to program the HDMItransmitter.

This section describes the local user interface, and in particular, theset of messages between the main board and the front panel controller,including the firmware design of the front panel controller. The term“front panel controller” refers to the following components:

LPC2103 ARM7 processor

Noritake MN11216A Vacuum Florescent Display

IR Receiver

IR Transmitter

Buttons physically attached to the front of the unit

LEDs visible on the front of the unit

These components may be located on the physical front panel board or themain board. They are all controlled by the LPC2103 processor and composethe “logic” front panel of the unit.

The front panel uses a simple control loop in the “main” function tocontrol the system. This loop checks for event indications from theInterrupt Service Routines (ISRs). The only hardware device that thisupdated outside an ISR is the vacuum florescent display. Display updatetiming is controlled by timer0 but the updates are carried out in theforeground.

These ISRs run independently of the main loop:

Timer0—display update timer

Timer1—Capture incoming IR data (external capture mode)

Timer2—Drive outgoing IR data (drive external match pin)

UART0—IXP455 communication

As shown in FIG. 22, UART interrupts are masked during display updates.The UART has a 16 byte deep FIFO which will prevent data loss while theinterrupts are masked. The timer capture and match interrupts supportingIR are not masked during display updates.

During normal operation without IR traffic the display update will blockfor 200 of every 500 micro seconds.

Flash updates need to be done at the end of a full display update toprevent the display from scrolling. A 256 byte flash update requires 1mS during which time all ISRs must be masked. Empirical testing showsthat a 2 mS delay between the end of one display update and the start ofthe next is not noticeable on the display. Blocking the UART ISRs for amillisecond may cause data lost depending the on the baud rate and howmany bytes are in the fifo when interrupts are masked. Blocking thetimer capture and mask interrupts for a millisecond will disrupt IRtraffic. The flash update can be held off until the IR transmitter isidle but there is no way to process received IR data during a flashwrite.

Inter-processor communication is shown in FIG. 23. It is done using aUART on the LPC2103 and another UART on the IXP455. The basic messageformat is:

-   -   Leader <STX>    -   Size    -   Command    -   Payload    -   Trailer <ETX>

The values of STX and ETX will be contained in some of the binary datainvolved in the IR messages. The message processor validates messages itremoves from the RX queue to correctly assemble messages.

The transmitter units contain a Noritake vacuum florescent display (VFD)on the front panel. The display characteristics are as follows:

-   -   Pixel based 112×16 VDF    -   Smallest updatable unit is a grid, 6×16 pixels    -   The maximum time to update the complete display is 10 mS per the        Noritake specification. We are currently updating the entire        display every 20 mS without any noticeable dimming or flicker in        the display.    -   Grid updates are only interrupted by timer 1 and 2 when the IR        is running in raw mode. These ISRs must be extremely fast.        Interrupting the display for to long can result in visible        problems.

The physical display is updated a single grid at a time. Grid updatesare controlled by timer0, match register 0, which expires and interruptsevery 500 uS. A grid update requires approximately 200 uS.

As shown in FIGS. 24 and 25, when timer 1 expires it signals the mainloop to begin a grid update. The match register reloads automaticallyeach time it expires so the only action required in the ISR is sendingthe signal and clearing the interrupt.

The IXP455 can update the front panel display by sending text strings toupdate part of the display or by sending a full frame buffer. Text basedupdates will be processed by the front panel processor and written tothe frame buffer in a 5×7 LCD font. Full frame buffer updates will notbe processed by the front panel; they will be displayed as received.

The front panel provides two frame buffers to the IXP455. The IXP455 maywrite to either frame buffer at any time. The IXP455 may direct thefront panel to change which frame buffer is used to update the physicaldisplay.

The IXP455 may update the front panel frame buffers by:

-   -   Sending a “Display Text String” command to update a portion of        the display.    -   Sending a “Full Display” command followed by a full update to        one of the frame buffers.

The front panel on the receiver unit contains five (5) LEDs in place ofthe VFD. During boot up of the IXP455 the LEDs will display an “active”pattern to indicate that the unit is alive. Once the Linux kernel hasbooted and the main application is running the LEDs will be controlledby the IXP455 via FP_LED messages.

The IR Subsystem of the local user interface involves receipt andtransmission protocols. IR will be received using a commercial IRreceiver, such as those marketed by Sharp Corporation. The receiverdemodulates the incoming signal and outputs a waveform representing thereceived signal.

The output signal from the receiver is connected to an external capturepin on one of the LPC2103's timers. The timer is configured to interrupton any edge transition on the capture pin. The time of the first edgetransition is not recorded but the timer itself will be reset. Onsubsequent edge transitions the timer value will be recorded and thetimer will be reset. The time values will be recorded until it isdetermined that an entire IR command has been received at which time thetimer values will be sent to the IXP455 in an “IR Received” message.

IR is transmitted using an IR LED. A timer running in match mode and anexternal GPIO pin is used to drive the LED. Upon receipt of an IRmessage from the man processor the GPIO pin enables the LED and thetimers match register will be loaded with the first time value from theIR message. When a match interrupt occurs the GPIO pin will toggle, viathe timer logic, and the match register will be loaded with the nextvalue from the IR message. This toggle and reload operation willcontinue until all timer values contained in the IR message have beenused.

The front panel provides the IXP455 with its (1) Firmware Version, (2)Hardware Version, (3) Processor ID, and (4) Boot Loader Version.

The infrared system provides an extension of infrared remote controlusing an IP based network. The method described extends the range of aninfrared (IR) remote control using a wired or wireless IP based network.Although, the method describes extending the range of an IR remotecontrol using an IP based network, the IR remote control could beextended using other types of networks.

Infrared remotes controls use infrared light to control electronicdevices. The devices that need to be controlled normally need to be inline of sight with the remote control and at a short distance from theremote, normally 15 feet. This means that an IR remote control will notwork with devices that are in another room, too far from the receiver orthat are behind obstructions.

Therefore the described method outlines a mechanism to solve theseproblems and to extent the range of an infrared remote control.

As shown in FIG. 36, the infrared data from the remote control isdetected by an infrared receiver that converts the infrared informationinto an electrical signal that a microcontroller can read. Themicrocontroller extracts the timing information. The timing informationis transmitted using a wired or wireless IP network to anothermicroprocessor that will use the timing data to reconstruct and retransmit the infrared data using an infrared LED. The microcontrollercan be substituted by a microprocessor, or a combination of amicrocontroller and a microprocessor.

With a fixed carrier infrared receiver, the infrared receiver convertsthe infrared light signal into an electrical signal that can be read bythe microcontroller. With a fixed carrier infrared receiver the carrieris always known. The carrier is removed from the incoming IR signal andonly the data is sent to the microprocessor. The data pin from the IRreceiver is connected to a general IO pin. The microprocessor will lookfor low to high and high to low transition in the data and measure thetime between those events. This timing data is packed and sent toanother microcontroller using an IP based network. The secondmicrocontroller decodes the data packets from the network and extractsthe timing data information. With the timing information themicrocontroller reconstructs the infrared data and adds the knowncarrier to it. The data with the carrier is sent to the infraredtransmitter circuit and to the electronic device to be controlled.Normally the infrared transmitter consists of an infrared LED and asmall transistor amplifier.

With a universal carrier Infrared Receiver, the method is similar to thefixed carrier, but the infrared receiver does not remove the carrier.The output from the IR receiver is connected to a general IO pin. Themicroprocessor will look for low to high and high to low transition inthe data and measure the time between those events. This timing data ispacked and sent to another microcontroller using an IP based network.The second microcontroller or process within the primary microcontrollerdecodes the data packets from the network and extracts the data andcarrier timing information. With the data and carrier timing informationthe microcontroller then reconstruct the data with the carrier. Thecarrier does not have to be added because it is embedded in the datareceived. The data with the carrier is sent to the infrared transmittercircuit and to the electronic device to be controlled.

With a universal carrier Infrared Receiver with carrier detect, themethod is again similar to the fixed carrier, but the infrared receivercomputes the carrier frequency and removes the carrier from the incomingdata. The infrared receiver extracts the carrier from the incoming IRsignal, computes the carrier frequency, sends the carrier informationand the data to the microcontroller. The microcontroller takes the datafrom the IR receiver and looks for low to high and high to lowtransition in the data and measure the time between these events. Thistiming data and carrier are packed and sent to another microcontrollerusing an IP based network. The second microcontroller decodes the datapackets from the network and extracts the data timing information andthe carrier. With the data timing information the microcontrollerreconstructs the infrared data and adds the carrier to it. The data withthe carrier is sent to the infrared transmitter circuit.

The system of FIG. 1 is a system where extending infrared remote controlsignals is particularly useful. In FIG. 1, video signals from thetransmitter are sent via a wireless IP network to receivers that areconnected to respective displays, this way video from the source can bedisplayed on multiple displays. Transmitter 11 and receivers 12 haveinfrared receivers and can receive control signals from an infraredremote. Thus, using the extension method described above infraredsignals can be received at the transmitter and transmitted to thereceivers via the IP network.

Now, the system's web user interface is described. The system provides aWeb User Interface System that allows the user to configure systemsettings; display hardware and firmware version, connection status andsignal strength etc; and update firmware. The web interface authorizes asingle user, e.g. an audio and video (AV) system integrator, toconfigure hardware and software settings of Tx and R×(s) via HTML Webpages. A Web browser communicates with embedded Web server using a10/100 Ethernet or a 802.11a link connected to either the Tx, R×(s)directly, through a router, or through web proxy via the Tx/Rx. TheEthernet link also transmits audio, video and control data.

FIG. 26 shows the basic blocks of the web interface.

Each Tx or Rx unit contains an embedded Web server. When the AV systemintegrator enters the URL of the IP address of a Tx or Rx, the embeddedWeb server in the Tx or Rx serves up the default page of the webinterface. The system integrator can then log in as the authorized user.Once the Web server authenticated the access, the system integratorinteracts with the system through HTML Web pages.

The web interface allows the AV system integrator to enter configurationparameters to configure the device through HTML Web pages. In addition,the system integrator can query hardware and firmware version as well asdevice status from the system. The Web pages therefore contain dynamiccontent. The system uses the Common Gateway Interface (CGI) standard toserve up dynamic Web pages.

The Web browser requests the URL corresponding to a CGI script orprogram, which follows the CGI protocol to retrieve input data from adynamic Web page as well as compose dynamic Web page content.

The web interface incorporates the following components:

-   -   An embedded Web server, such as is available by open source        under the name “Boa Web Server;”    -   HTML Web pages    -   CGI programs that use Common Gateway Interface (CGI) standard to        interface with the AV core system software. The following        websites provide basic information on CGI programs:        -   http://cgi.resourceindex.com/Documentation and        -   http://hoohoo.ncsa.uiuc.edu/cgi

The design of WEBUIS should abstract interfaces between CGI programs andthe AV core system so that when we expand Web pages and add CGIprograms, the interface remain unchanged.

The Flash Memory Strategy is now described.

The flash memory map is split up into 3 regions, shown in FIG. 27:

-   -   The Boot region, which contains the boot loader firmware;    -   The compressed kernel region; and    -   The compressed image of the CRAMFS region, which contains the        application, modules, libraries, FPGA bit stream and utilities.

Persistent storage is required for the kernel, application, front paneland FPGA update files and configuration. Additionally, the receiverupgrade package must be stored to provide for the ability to upgradereceivers from the transmitter.

The kernel is simply stored compressed in flash. When booted, the kernelextracts an initial CRAMFS (initrd) image from flash for use as its rootfile system. The CRAMFS image is stored as a compressed ext2 filesystem. The root file system contains the application, utilities,libraries and required update files.

Application upgrades are handled by downloading the compressed CRAMFSimage which is then written to FLASH by the application.

Boot itself may be field upgradeable if an application containing bootis downloaded to the appliance and the application reprograms the bootsector.

Dynamic memory allocation is needed for creating objects and eventsduring runtime. Dynamic memory allocation from the heap is limited toboot-up initialization, not only to prevent potential memory leaks, butalso to keep code deterministic in terms of memory usage.

The framework provides a memory pool class that may be utilized by anypart of the system that requires objects to be created/destroyeddynamically. A memory pool is nothing more than a fixed number of fixedsized blocks set aside at system start. When an object is created, oneof the blocks is used for the object. By knowing the type and maximumnumber of all objects in the system we can predetermine the memoryrequirements for the system and we can guarantee memory for each object.

The Video Packet Formats are now described. As shown in FIG. 28, a videoframe consists of an A/V subsystem Video Header, an ADV202 Header(assuming an ADV202 Raw Format output), a JPEG2000 Header, attributedata and compressed video data.

To avoid IP fragmentation and the associated performance degradation,video frames are packetized into RTP packets such that each can fit intoa single IP packet. Since for each packet, an RTP header, UDP header,IPSec ESP header and IP header will be added, the maximize size ofpayload in each RTP packet will be equal to the MTU of the network (1500bytes) minus the total length of those headers which is to bedetermined.

A. A/V Subsystem Video Header is shown in FIG. 29. It is generated bythe A/V subsystem FPGA.

The ADV202 will insert a ADV202 specific header at the beginning of thecode stream. FIG. 30 contains information about this ADV202 header.

The JPEG2000 Header is shown in FIG. 31. Information about JPEG2000markers can be found in the ISO/IEC15444-1 standard. The JPEG2000compliant header contains main and tile headers from the JPEG2000standard. It consists of all the parameters needed to decode an imagecorrectly, including the quantization stepsizes. FIG. 31 lists the mostcommon markers that are inserted into the compressed codestream by theADV202.

The A/V subsystem FPGA will append an Audio Header to an audio frame toallow the receiver to synchronize the audio with the correct videoframe. Like video frame, the audio frame is packetized into RTP packetssuch that each fits in an IP packet. FIG. 32 shows an example audioframe format.

The Audio Header of FIG. 32 contains the fields shown in FIG. 33.

Next, we describe playback synchronization and error control models.

A. Adaptive Synchronization Algorithm

The adaptive synchronization algorithm uses minimal knowledge of networktraffic characteristics. The algorithm is immune to clock offset anddrift between the transmitter clock and the receiver clock while itensures the QoS in terms of end-to-end delay, delay jitter, and lossratio.

The details of the algorithm are summarized here, for a full explanationof the implementation see IEEE journal Vol. 14 No. 7 “MultipointMultimedia Teleconference System with Adaptive Synchronization”

Instead of having a fixed playout point the application is allowed toadjust it depending on network conditions. This means fewer packets arediscarded because they arrive late. Also instead of discarding all datathat arrives late we allow packets that arrive only slightly late to beplayed back. This adds a small amount of distortion but is better thanmissing data.

The synchronization scheme requires the user to specify the maximumacceptable jitter, JMax and the maximum acceptable packet loss ratiocaused by synchronization measures, LMax. At the transmitter each packetcarries a timestamp ti,g, indicating its generation time. At thereceiver a Playback clock (PBC) and three event counters, namely thewait counter Cw, the nonwait counter Cnw, and the discard counter Cd,with associated thresholds Tw, Tnw, and Td, respectively, aremaintained.

The PBC is nothing but a virtual clock at the receiver which emulatesthe clock at the sender. The motivation to have the PBC is that once thesource clock can be reproduced at the sink, the synchronization problemmay be readily solved. At the receiver, the PBC is initiated accordingto the time stamp carried by the first received object, updated by thereceiver clock, and adjusted based on the contents of the threecounters. The vicinity of a packet's arrival time in reference to thePBC time is partitioned by the wait boundary (Bw,) and discard boundary(Bd) in to three regions: the wait region, the nonwait region, and thediscard region, shown in FIG. 34.

The arrival time ti,ar, in reference to the PBC, of the ith packet mayfall into one of the three regions with respect to its associated twoboundaries. The synchronization algorithm conforms to the followingrules.

-   -   1. If the packet with time stamp tz,g arrives before Bi,w        (within the wait region), then it will be played back at Bi,w        (waiting until the wait boundary).    -   2. If the packet with time stamp ti,g arrives after Bi,w but        before Bz,d (within the nonwait region), then it is played back        immediately.    -   3. If the packet with time stamp tz,g arrives after Bid (within        the discard region), then it is discarded.

The PBC is synchronized to the transmitter clock using the followingalgorithm.

-   -   1. Upon receiving the first successfully arrived packet set the        initial PBC time equal to the time stamp carried by this packet.    -   2. Upon receiving the ith packet, compare its time stamp ti,g        with its arrival time ti,ar, (the current PBC time PBC(t)): if        ti,g>ti,ar, increase the wait counter by one and do not playback        the object until Bw; else if ti,g<ti,g+Emax, increase the        nonwait counter by one and playback the packet immediately;        otherwise (i.e., ti,ar≧ti,g+Emax) increase the discard counter        by one and discard the packet.    -   3. Check the most recently increased counter: if overflows        continue; otherwise go to Step 2.    -   4. When the nonwait counter or the discard counter overflows: if        the wait counter is not full, decrease the PBC:

PBC(t)=PBC(t)−Δ

-   -   -   and go to Step 5; otherwise go to Step 5. When the wait            counter overflows: if the nonwait counter is not full.            increase the PBC:

PBC(t)=PBC(t)+Δ

-   -   5. Reset all counters and go to Step 2.

The thresholds of the three counters shown in FIG. 34 are critical tothe performance of the synchronization algorithm. In particular, thethreshold of wait counter, Tw, governs how sensitive the synchronizationscheme is to the network improvement. Its value should span a timeinterval of at least the order of ten seconds. Otherwise thesynchronization would be too sensitive to network improvement;consequently too frequent down shift of the PBC would likely occur.

B. Inter-Stream Synchronization

When inter-stream synchronization is needed a Group (Playback Clock) PBCis required. The Group PBC is set to the slowest of all PBC's. ThisGroup PBC dominates the playback of all media in the synchronizationgroup. Meanwhile, each medium in the intermedia synchronization does itsown synchronization as if it was not in the group, but the discarddecision is made in reference to the Group PBC.

An example of a group playback clock schematic is shown in FIG. 35.

C. Error Control

There are a number of characteristics of continuous media streams suchas audio and video.

-   -   Strict timing requirements—If the data is not delivered before a        certain point in time it has to be discarded.    -   Some tolerance of loss—The amount of loss that can be tolerated        depends on the medium, the encoding techniques and human        perception.    -   Periodicity—Video or audio should be delivered at a fixed rate.        When transmitting continuous data across a network this        periodicity is normally lost.

Data transmitted across networks are normally subject to delay, delayjitter, resequencing of packets, and loss of packets.

The RTP protocol uses the packet sequence number to reorder packets in astream. Buffering packets at the receiver overcomes problems related tonetwork delay jitter. However as RTP uses UDP multicasting to delivercontinuous video and audio streams packet loss will occur.

There are several methods for dealing with packet loss of video or audiodata in order to provide and acceptable quality of service (QoS).

Automatic Repeat Request (ARQ)

Forward Error Correction (FEC)

Hybrid Error Control (ARQ/FEC)

Interleaving

Error Concealment

Using ARQ a lost packet will be retransmitted by the sender. Loss ofdata can be detected by the sender or by the receiver. Detection by thesender requires that every receiver send an ACK for each receivedpacket. Clearly when multicasting to a number of receivers this consumessignificant bandwidth. Detection by the receiver is more efficient inthis case. The receiver sends a NAK if a packet sequence number ismissed in the stream. If all receiver's miss the same packet this canresult multiple NAK's being sent to the sender for the same packet. Thiscan be avoided by multicasting the NAK instead of unicasting so otherreceivers in the group realize a packet has already been re-requested.

FEC transmits, with the original data, some redundant data, calledparities to allow reconstruction of lost packets at the receiver. Theredundant data is derived from the original data using Reed-Solomoncodes or a scheme which uses the XOR operation. The FEC transmittersends k packets along with h redundant parity packets. Unless thenetwork drops>h of the h+k packets sent, the receiver can reconstructthe original k information packets. RFC 2733 specifies a RTP payloadformat for generic forward error correction.

The disadvantage of FEC is that the redundant data consumes bandwidthand the difficulty in choosing the right amount of redundancy forvarious network conditions. A solution to this is to send redundant datawhen a retransmission is required instead of the original packet. Thisis known as Hybrid Error Control.

With ARQ and ARQ/FEC it is important that the retransmitted data orparity packet is received before the playout point otherwise the packetwill be discarded. This requires having a jitter buffer at the receiverlarge enough to provide a delay equal to the network delay plusretransmission of a lost packet. The strict delay requirements ofinteractive systems usually eliminate the possibility ofretransmissions. However in a non-interactive system such as ours aplayout delay of 500 ms-1 second should not cause a problem.

When the packet size is smaller than the frame size and end-to-end delayis unimportant, interleaving is a useful technique for reducing theeffects of loss. Packets are re-sequenced before transmission so thatoriginally adjacent packets are separated by a guaranteed distance inthe transmitted stream and returned to there original order at thereceiver. Interleaving disperses the effect of packet losses. Multiplesmall gaps in the reconstructed stream are less noticeable to the userthan a large gap which would occur in a non-interleaved stream.

Techniques for error concealment may be used by the receiver and notrequire assistance from the sender. These techniques are useful whensender based recovery schemes fail to correct all loss, or when thesender of a stream is unable to participate in the recovery. Errorconcealment schemes rely on producing a replacement for a lost packetwhich is similar to the original. Insertion-based schemes are thesimplest to implement and repair losses by inserting a fill-in packet.This fill-in for audio data is usually very simple either silence, whitenoise or the repetition of the previous packet. Silence or noiseinsertions have poor performance however repetition or repetition withfading is a good compromise when compared to the more complexregenerative concealment methods.

When the system is non-interactive and the transmission is multicast,latency is less important than quality. Bandwidth efficiency is aconcern as the transmission link may be wireless. Interleaving is seento be an effective way of reducing the effect of packet loss coupledwith error concealment by repeating the previous packet when a packet islost.

If interleaving and error concealment do not provide acceptable QoS anARQ or ARQ/FEC scheme can be substituted.

We now describe the hardware design of the main board. The Main board'spurpose is to be the main platform for both the transmitter 11 andreceivers 12. This dual purpose for the board is accomplished byproviding an expansion connector. The expansion connector will be usedto connect a HDMI receiver for the transmitter and a HDMI transmitterfor the receiver.

The Main board also provides for the addition of a ZIgbee wirelessconnection, in order to allow easy control of the appliance. An examplemain board is shown in FIG. 37, and in this example is based on theIntel XSCALE IXP455 processor running at 266 Mhz. The following featuresof the IXP455 are implemented in order to complete the board:

-   -   32 Mbytes of Flash    -   64 Mbtes of DDR memory running at 266 Mhz(133 Mhz clock)        Expandable to 128 Mbytes    -   1 Mini-PCI slot(Type 3 connector, 3.3V compatible only) running        at 33 Mhz with option for second PCI slot is for wireless radio    -   1 10/100 Ethernet port    -   1 Front Panel Serial connection    -   1 Linux debug port    -   1 USB 2.0 compliant host port    -   1 USB 1.1 compliant device port    -   I2C bus    -   SPI Port

In addition to the features directly tied to the IXP455 processorperipherals the following functions are implemented in order to completethe system.

-   -   ADV202 JPEG2000 Codecs for real time video compression of up to        1080i    -   ADV202 to PCI interface implemented inside of an FPGA to allow        reuse of video section in multiple platforms    -   I2S. SPDIF and Audio AD/DA interface via FPGA    -   Serial Port for Serial Pass through    -   Serial port for Zigbee expansion board    -   Serial port for to interface to IR remote controller    -   DMA engine to move compressed video and audio from codecs into        and out of system memory    -   Second I2C bus implemented in FPGA to allow the buffered access        to video chip configuration and audio

The details of the memory map for the IXP455 processor can be found intwo documents, Intel IXP45X and Intel IXP46X Product Line of NetworkProcessors Developer's Manual (referenced above) and Intel IXP45X andIntel IXP46X Product Line of Network Processors Data Sheet (alsoreferenced above), and will not be repeated herein. Some of the addressspaces contain individual control registers.

The memory controller takes care of the fact that most devices on thebus are 32 bits wide, For example one word contains address's zerothrough three. The processor is Big Endian Data so the most significantbyte of a thirty two bit word is stored in memory first followed by thesecond. For example if the following value was read from the PCI Bus0xFFEEDDCC and stored in DDR starting at location 0x100, then FF wouldbe stored at 0x1000, followed by EE at 0x1001, DD at 0x1002 and CC at0x1003.

The FPGA memory map will be the same as the FPGA, in terms of the ADV202codec's, will be the same as the PCI card.

The processor supports DDR1 266 Mhz (133 Mhz clock) The device isconfigurable for 128 Mbit, 256 Mbit, 512 Mbit and 1 Gbit DDR memory aslong as they are partitioned on 32 Mbyte boundaries.

FIG. 38 details how the DDR memory fits into the memory map of theprocessor as designed on the main board.

The Expansion Bus memory is mapped for flash, expansion card i/o, andFPGA parallel programming. FIG. 39 shows the expansion bus flash memoryconfiguration.

Expansion Card I/O memory usage is shown in FIG. 40. Twelve databits(0..11), CS_N, RD_N, WR_N and ALE go to the expansion bus connectorgiving a space of 4K bytes of memory mapped space for I/O expansion. Theparticular chip select used must be configured in Multiplexed addressand data mode. Twelve address bits are latched on the falling edge ofALE after which Data lines 0 thought 7 are available for eight bit readsand writes This address space is for future use on projects whereswitching logic is required, or for future projects.

The expansion bus of the IXP455 is a general purpose bus broken intoeight chip selects that each cover a 32 Mbyte chunk of memory. Each chipselect is programmable in terms of its timing, data width, multiplexedor non multiplexed address and data. The following tables show how toset up the expansion bus for the Expansion Connector and to program theFPGA.

Chip select 0 flash is set up by the Boot Configuration register andRedboot. The Chip select for the Expansion connector and the FPGA writesshould be set up as shown in FIGS. 42 and 43. FIG. 42 shows the set upfor the Expansion bus from the IXP455 to the Expansion Connector andFIG. 43 shows the set up for the Expansion bus from the IXP455 to theFPGA programming.

Each chip select is individually configurable so that different memoryspaces on the expansion bus may have different data widths and timing.The Timing and control register for Chip Select 0 is shown below theTiming and Control Registers for the other seven chip selects areessentially identical.

The IXP455 is set up as the arbiter on the PCI Bus. There are threeslots on the PCI bus as detailed in FIG. 44. The PCI bus uses a standard32 bit, 33 Mhz, Mini PCI interface and signals.

The processor implements one Ethernet port. The Ethernet port is usesNPEA B. NPEC must also be enabled in order for the Encryption engine tobe enabled. The Ethernet PHI is connected Via a MII interface to theIXP455. The Ethernet port is strapped to address 0, It will support10/100 full or half duplex with auto negotiation. The I/O lines on NPECmust be pulled up, in order for the MII controller to operate properly.NPEA must be soft disabled by writing a one bit 11 and bit 19 of theEXP_UNIT_FUSE_RESET register.

The board supports four serial ports. Two serial ports are available viathe processor and two are available across the PCI bus via the PCIinterface as detailed in FIG. 45.

The USB host controller supports the EHCI register Interface, HostFunction, Low Speed Interface, and Full Speed Interface. The signalinglevels are compliant with the 2.0 specification.

The USB controller on the IXP455 supports USB 1.1 Low and Full speed,however the board pull ups are enabled for Full speed mode. Signalinglevels are also compliant with the USB 1.1 specification.

An 8K (64K bit) Serial EEPROM is connected to the IXP455 I2C bus. It ishard wired to address seven. The device supports byte and 32 bit pagewrites.

A rechargeable battery is provided to maintain at least 100 hours ofreal time clock after main board power down. During power down, the Realtime clock draws 10 uA max on the VBAT pin for the battery.

The Main board uses a programmable clock synthesizer to produce theclocks required for the processor, PCI, DDR, Expansion bus, ADV202's,FPGA, and Ethernet. The programmable clock synthesizer has 8 differentregisters that allow a different set of clock frequencies to be produceddepending of the state three configuration inputs to the chip. Theseregisters are set in the design by a set of external resistors. Theclock synthesizer also produces a spread spectrum clock with a −2% 34Khz modulation on the processor and PCI clock in order to reduceemissions. Using the configuration registers of the Clock synthesizer,the spread spectrum may be turned on or off. The input to thesynthesizer is a 25 Mhz clock produced by an external oscillator. FIG.46 shows the configuration of the clock synthesizers registers.

The main board clock register setup is shown in FIG. 47.

FIG. 48 defines the processor GPIO pins on the IXP455 and theirfunctions.

The IXP455 processor has many options that are only available at boottime. The processor reads the values on the expansion bus address linesduring boot in order to determine how these options are set. The addresslines are internally pulled up with 47 KOHM resistors. If the addresspin is left floating the processor will read a value of one on theaddress line. Any line that needs to a zero at boot is pulled down witha 4.7 Kohm resistor. The options are read into Configuration Register 0.FIG. 49 shows the boot configuration as defined on the main board. FIG.50 shows the user defined values board revision. New Revisions can beadded if software flow needs to change, i.e., there is a change inmemory size etc.

Continuing with the hardware schematic of FIG. 37, each of the ADV202parts will have a combined address and data bus. Each bus contains thefollowing signal sets. Indicated directions are relative to the FPGA.

Data[31:0]—Bidirectional

Address[3:0]—Output

CS_n, WE_n, RD_n, DACK_n*—Outputs

INTR_n, ACK_n*, DREQ_n*—Inputs *These signals are used by the ADV202 tonegotiate DMA transactions. The FPGA assumes worst-case bus timing andpulls/pushes data from the ADV202s as fast as possible.

Another signal, SCOMM5, will be a single FPGA output that will be routedto both ADV202s scomm pins.

Expansion Support is via the expansion board connections of FIG. 37.Those connections are shown in detail in FIGS. 51A and 51B.

This now describes the so-called Avocent Audio Visual Protocol (AAVP).It is intended for the communication among audio visual products. It isused to establish and manage an IP-based network for the transmission ofreal-time audio-visual data over a wired or wireless medium. Instead ofbeing a monolithic protocol, AAVP is a collection of related protocolsorganized in planes and layers.

FIG. 52 illustrates the model of this protocol. The transmitter shown isthe source of audio-visual data generated from the program source. Thereceiver is the sync of such data. A transmitter and one or morereceivers form a network. This protocol describes the messages exchangedbetween the transmitter and receivers over the network.

The functions of this protocol can be divided into Control Plane andData Plane. In the Control Plane, there are four types of messages,namely REQ (Request), CFM (Confirm), IND (Indication) and RESP(Response). A receiver sends REQ to the transmitter which responds withCFM. Reversely, the transmitter can send IND to a receiver which in somecases responds with RESP. The purpose of the Control Plane is toestablish real-time transport sessions in the Data Plane to transferdata packets across the network.

The physical medium of the network can be wired (e.g. Ethernet) orwireless (e.g. 802.11a).

This section explains the functions of Control Plane and Data Plane. TheControl Plane is further divided into Link Control and Session Control.Each plane is implemented as multiple layers. The concept of planes andlayers is illustrated in FIG. 53.

1. Control Plane

A. Link Control

-   -   This establishes the communication link between a transmitter        and a receiver. It enables the transmitter and receiver to        communicate with each other. The protocol used for this is        called Avocent Link Initiation Protocol (ALIP) and is described        in more detail below. A receiver probes for any transmitters by        sending a probe request message. A transmitter responds with a        probe confirm message. The receiver then joins the network by        sending a join request message. The transmitter can accept or        deny the request based on MAC address filtering. The transmitter        may assign an IP address to the receiver if the receiver does        not have a manual IP address. The above process applies to both        wired and wireless media. For wired medium, this is        straight-forward. For wireless medium, it may involve scanning        different radio channels for a transmitter.

B. Session Control

-   -   This establishes an audio-visual session between a transmitter        and a receiver. It enables the receiver to receive and play back        audio-visual data coming from the transmitter. The protocol used        for this is called Avocent Session Initiation Protocol (ASIP)        and is described in greater detail below. The transmitter        informs the receiver of the UDP ports used by the RTP streams,        the multicast address, as well as the audio/video parameters.

2. Data Plane

-   -   This transfers RTP and RTCP packets to convey real-time        audio-visual data. The RTP protocol is extended to support        reliable multicast which may be necessary for audio data as        audio packet loss is more perceptible.

The link protocol: ALIP

ALIP is used to establish and maintain the communication link between atransmitter and a receiver in a network. It also passes control andinformation messages between transmitter and receivers. If there aremultiple receivers, there is one ALIP link between the transmitter andeach receiver. The concept of ALIP links is illustrated in FIG. 54. SomeALIP messages are carried by UDP packets since they may be broadcastbefore an IP address is assigned.

The ALIP message format is as follows. Each ALIP message is made up of aheader and a payload. The payload follows immediately after the header.All multi-byte fields are transmitted in network byte order(big-endian). The message header has a fixed length of 12 bytes andconsists of the fields shown in FIG. 55.

The message payload contains fields specific to the message type. Theyare listed in the next section. Type codes are shown in the parenthesesfollowing the type names. The status codes are shown in FIG. 56 and areused in various ALIP messages to indicate either a reason for a requestor the failure reason in a response.

1. Message Type: ALIP_PROBE_REQ (0x0001)

A receiver broadcasts this message to probe for a transmitter. On awireless network, it uses this message to discover a transmitter on aparticular radio channel. If no response is received within the timeoutperiod ALIP_PROBE_REQ_TO, it retries on the same radio channel forALIP_PROBE_RETRY_CNT time(s). When retries fail, it scans the next radiochannel. On a wired network, it always retries on the same physicalmedium. This message is broadcast because the receiver does not know theIP address of a transmitter yet.ALIP_PROBE_REQ_TO=200 ms (or other suitable timing)

ALIP_PROBE_RETRY_CNT=2

Destination IP address=255.255.255.255

2. Message Type: ALIP_PROBE_CFM (0x0002)

A transmitter broadcasts this message in response to ALIP_PROBE_REQ. Itallows a receiver to learn about the presence and properties of thetransmitter. The parameters of this message are shown in FIG. 57. It isbroadcasted because the transmitter does not know the IP address of thereceiver.

Destination IP address=255.255.255.255

3. Message Type: ALIP_JOIN_REQ (0x0003)

The configuration of this message type is shown in FIG. 58. A receiverbroadcasts this message to request to join the network. This message isbroadcasted because the receiver may not have an IP address yet. If noresponse is received within the timeout period ALIP_JOIN_REQ_TO, itretries for ALIP_JOIN_RETRY_CNT time(s). When retries fail, it regardsthe request failed.

ALIP_JOIN_REQ_TO=200 ms (or other suitable timing)

ALIP_JOIN_RETRY_CNT=2

Destination IP address=255.255.255.255

4. Message Type: ALIP_JOIN_CFM (0x0004)

The configuration of this message type is shown in FIG. 59. Atransmitter broadcasts this message in response to ALIP_JOIN_REQ. Thismessage is broadcast because the receiver may not have an IP addressyet.

Destination IP address=255.255.255.255

5. Message Type: ALIP_POLL_REQ (0x0005)

The configuration of this message type is shown in FIG. 60. A receiversends this message periodically (once every ALIP_POLL_PERIOD) to pollthe transmitter in the network it has joined. If it does not receive aresponse within the timeout period ALIP_POLL_REQ_TO, it retries forALIP_POLL_RETRY_CNT time(s). If a response is still not received afterretries, the receiver detaches itself from the network.

On the other hand, a transmitter uses this message to check if areceiver previously joined the network is active. If it does not receivethis message from a receiver once in the period 2*ALIP_POLL_PERIOD, itdetaches that receiver from the network.

ALIP_POLL_PERIOD=2 s (or other suitable timing)ALIP_POLL_TIMEOUT=200 ms (or other suitable timing)ALIP_POLL_RETRY_CNT=2 (or other suitable count)Destination IP address=Transmitter IP address

6. Message Type: ALIP_POLL_CFM (0x0006)

The configuration of this message type is shown in FIG. 61. Atransmitter sends this message as a response to ALIP_POLL_REQ from areceiver.

Destination IP address=Receiver IP address

Using the above message types, a normal link establishment is shown inFIG. 62.

A scenario where the transmitter provides no response to a probe requestis shown in FIG. 63.

A scenario where the transmitter provides no response to a poll requestis shown in FIG. 64.

A scenario where the receiver provides no poll request is shown in FIG.65.

A scenario where a join request fails because a receiver MAC address isnot approved is shown in FIG. 66.

This now describes the so-called Avocent Session Initiation Protocol(ASIP). ASIP is used to establish an audio-visual session between atransmitter and a receiver. It enables the receiver to receive and playback audio-visual data coming from the transmitter. If there aremultiple receivers, there is one ASIP session between the transmitterand each receiver. The concept of ASIP sessions is illustrated in FIG.67. ASIP messages are carried by TCP streams which provide reliabilitytransport. TCP is possible because ASIP messages are unicast and the IPaddresses of the transmitter and receiver in a session are known to eachother (with thanks to ALIP).

The ASIP message format is as follows. Each ASIP message is made up of aheader and a payload. The payload follows immediately after the header.All multi-byte fields are transmitted in network byte order(big-endian). The message header has a fixed length of 12 bytes andconsists of the fields shown in FIG. 68.

The message payload contains fields specific to the message type. Theyare listed in the next section. Type codes are shown in the parenthesesfollowing the type names. The status codes are shown in FIG. 69 and areused in various ASIP messages to indicate either a reason for a requestor the failure reason in a response.

1. Message Type: ASIP_INIT_SESS_REQ (0x001)

The configuration of this message type is shown in FIG. 70. A receiversends this message to initiate a session to play the default programsource. If it does not receive the response within the timeout periodASIP_INIT_SESS REQ_TO, it regards the request failed. After requestfailure, the retry strategy is application specific.

ASIP_INIT_SESS_REQ_TO=1 s

Destination IP address=Transmitter IP address

The format of the 128-byte “EDID” field is defined in Section 3.1 (page9 of 32) of “VESA Enhanced Extended Display Identification Data StandardRelease A, Revision 1 Feb. 9, 2000” published by VESA (Video ElectronicsStandards Association).

The format of the 128-byte “EDID Extension” is defined in Section 7.5(page 79 of 117) of “A DTV Profile for Uncompressed High Speed DigitalInterfaces EIA/CEA-861-B May 2002” published by CEA (ConsumerElectronics Association) and EIA (Electronic Industries Alliance).

2. Message Type: ASIP_INIT_SESS_CFM (0x0002)

The configuration of this message type is shown in FIG. 71. Atransmitter sends this message to a receiver in response toASP_INIT_SESS_REQ. It carries time synchronization data, audio/videoparameters and RTP settings for the receiver to configure itself forplay-back. If not already started, play-back starts immediately afterthis message has been sent.

The format of the 15-byte AVI is defined in Section 6.1.3 (page 60 of117) of “A DTV Profile for Uncompressed High Speed Digital InterfacesEIA/CEA-861-B May 2002” published by CEA and EIA. For reference, it isalso listed in FIG. 72.

The format of the 12-byte AAI is defined in Section 6.3 (page 65 of 117)of “A DTV Profile for Uncompressed High Speed Digital InterfacesEIA/CEA-861-B May 2002” published by CEA and EIA. For reference, it isalso listed in FIG. 73.

3. Message Type: ASIP_TEARDOWN_REQ (0x0003)

The configuration of this message type is shown in FIG. 74. A receiversends this message to tear down an established ASIP session. If noresponse is received from the transmitter within the periodASIP_TEARDOWN_REQ_TO, it regards the request successful.

4. Message Type: ASIP_TEARDOWN_CFM (0x0004)

The configuration of this message type is shown in FIG. 75. Atransmitter sends this message in response to ASIP_TEARDOWN_REQ. Afterreceiving a teardown request, the transmitter removes the receiver fromthe active list. If there are no more active receivers, it stops theencoding hardware.

5. Message Type: ASIP_TEARDOWN_IND (0x0005)

The configuration of this message type is shown in FIG. 76. Atransmitter sends this message to forcefully teardown a session with areceiver.

6. Message Type: ASIP_ANNOUNCE_IND (0x0006)

A transmitter sends this message to notify a receiver that an A/V sourcehas become available. It awakes the receiver if it has entered the sleepstate so that it may initiate a session again.

7. Message Type: ASIP_AUTH_REQ (0x0007)

A transmitter sends this message to request for HDCP authentication dataof a receiver.

8. Message Type: ASIP_AUTH_CFM (0x0008)

The configuration of this message type is shown in FIG. 77. A receiversends this message in response to ASIP_AUTH_REQ. It contains HDCPauthentication data of the receiver. In case the receiver is an HDMIrepeater, this message contains the authentication data of all HDMIsinks attached to it.

9. Message Type: ASIP_SDP_IND (0x0009)

The configuration of this message type is shown in FIG. 78. Atransmitter sends this message to notify a receiver of the SourceProduct Description of the source.

The 28-byte SPD contains the fields shown in FIG. 79.

10. Message Type: ASIP_ISCR1_IND (0x000A)

The configuration of this message type is shown in FIG. 80. Atransmitter sends this message to notify a receiver of the InternationalRecording Code 1 of the current track.

The 18-byte ISRC1 contains the fields shown in FIG. 81.

11. Message Type: ASIP_ISCR2_ND (0x000B)

The configuration of this message type is shown in FIG. 82. Atransmitter sends this message to notify a receiver of the InternationalRecording Code 2 of the current track.

The 18-byte ISRC1 contains the fields shown in FIG. 83.

12. Message Type: ASIP_ACP_IND (0x000C)

The configuration of this message type is shown in FIG. 84. Atransmitter sends this message to notify a receiver of the Audio ContentProtection of the source.

The 30-byte ACP contains the fields shown in FIG. 85.

13. Message Type: ASIP_AVMUTE_IND (0x000D)

Using the above message types, a normal session establishment and teardown is shown in FIG. 87.

A scenario where there was authentication failure is shown in FIG. 88.

A scenario where a new session is established due to changes in thesource is shown in FIG. 89.

A scenario where there is an interruption of the program source is shownin FIG. 65.

A scenario showing control messages in a session is shown in FIG. 91.

This now describes the real-time transport protocol (RTP) and real timetransport control protocol (RTCP) employed by the present system 10. AnRTP session is an association among a set of participants communicatingwith RTP. While RTP carries real-time data across the network, RTCPmonitors the quality of service and conveys information about theparticipants in an on-going RTP session.

In this protocol, two RTP sessions are used, with one for audio data andthe other for video data. All participants in each RTP sessioncommunicate using a multicast IP address and two adjacent UDP ports. Theeven (2n) port is used by RTP and the odd (2n+1) port is used by RTCP.The purpose of the Control Plane (ALIP and ASIP) is to establish theaudio and video RTP sessions in the Data Plane. The concept of RTPsessions is illustrated in FIG. 92.

The format of the RTP header customized for this protocol is shown inFIG. 93 and the fields for the header are explained in FIG. 94.

Some custom RTCP packets are employed as well.

1. Custom Packet: Synchronization

To synchronize the clocks between a transmitter and a receiver, the RTCPpacket shown in FIG. 95 is used. A description of the fields is shown inFIG. 96. We define a custom packet instead of using the pre-definedSender Report because the one we define is smaller and more efficient toprocess. The packet type (PT) of the custom RTCP packets is 205 and thesubtype for Sync is 0.

2. Custom Packet: Retransmission

In order to support reliable multicast in RTP, the custom RTCP packetsshown in FIG. 97 are defined. An explanation of the fields for SingleNACKs is shown in FIG. 98 and for Sequential NACKs in FIG. 99. Thepacket type (PT) of the custom RTCP packets is 205 and the subtypes forSingle NACKs and Sequential NACKs are 1 and 2 respectively.Collectively, they are called NACKs.

While the invention has been described in connection with what ispresently considered to be the most practical and preferred embodiment,it is to be understood that the invention is not to be limited to thedisclosed embodiment, but on the contrary, is intended to cover variousmodifications and equivalent arrangements included within the spirit andscope of the appended claims.

1. A transmission system with a transmitter and multiple receiverscomprising: a transmitter communicating common audiovisual informationto active ones of the multiple receivers by point to multipointretransmission sessions between the transmitter and the respectiveactive receivers and awaiting newly receiving ones of the multiplereceivers, the receivers generally synchronizing their receipt of theaudiovisual information to the transmitter; the newly receiving ones ofthe multiple receivers broadcasting an retransmission message probingfor the transmitter when each newly receiving one of the multiplereceivers is prepared for receipt of the audiovisual information, thetransmitter responding to said probe with a point to point message tothe respective newly receiving ones of the multiple receiversidentifying the presence and properties of the transmitter andthereafter communicating the common audiovisual information to the newlyreceiving ones of the multiple receivers via a point to multipointretransmission communication protocol.
 2. A transmission systemaccording to claim 1, wherein the newly receiving ones of the multiplereceivers send requests to join the point to multipoint retransmissionsessions from the transmitter.
 3. A transmission system according toclaim 1, wherein the transmitter periodically sends a point to pointpolling message to the active ones of the multiple receivers to querywhether the respective active ones remain active.